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00 Report To The Honorable Paul Simon 
12 United States Senate 



Pell Grant Validation Imposes Some Costs 
And Does Not Greatly Reduce Award Errors: 
New Strategies Are Needed 



In 1982-83, in an effort to reduce the continuing problem of error in awarding Pell 
grants for postsecondary education, the Department of Education increased to 1 ,66 
million the number of applicants who must document or "validate" their eligibility. 
This increased validation imposed some costs and burdens on the schools and had 
some impacts on students, although the estimated cost to institutions was less than 
1 percent of the total Pell program, which provided $2.4 billion in grants in the 
1982-83 school year. This smaller-than-1-percent,cbst (about $23 million) was not, 
however, offset, since only about $22.million was clearly saved. 

The Department's studies, while lim^.^d in some respects,' identify continuing 
problems with award accuracy. The error is sizable: underawards and overawards, 
totaled an estimated $649 million in 1982-83, despite the increased validation. The 
error is also persistent: the proportion of cases with student error has not decreased. 
Further, the Department's policy focused on student error and on overawards rather 
than on both institutional and student error and on both overawards and under- , 
awards. 

A review of Department policies and procedures shows that improvements are 
needed in specifying error-reduction goals, developing and testing a broader set of 
strategies to meet these goals, coordinating the management of error-reduction 
efforts, and evaluating their effects. 
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The Honorable Paul Simon 
United States Senats 

Dear Senator Simon: 

In your letter of September 7, 1983, you asked that we ana- 
lyze the efforts of the Department of Education to verify data 
submitted by applicants to the Pell grant program* The Depart- 
ment's policy or methodology is called "validation," This report 
presents data we gathered to respond to your questions about the 
effects of validation on institutions of higher education which 
carry out the procedure and on student applicants for financial 
aid. We also discuss the Department's data showing a continuing 
problem of error in the grants awarded under this program, our 
evaluation of the soundness of those data, and our observations 
on the goals ^ strategies, and management of the Department's 
overall reponse to error in the Pell program* 

As we arranged with your office, we are sending copies of 
this report to interested congressional committees, to the U,S, 
Secretary of Education, to the Acting Director, Office of Manage- 
ment and Budget, and to members of the higher education communi- 
^y* Copies will be made available to others who request them. 



Sincerely yours. 




Eleanor Chelimsky 
Director 
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GENERAL ACCOUNTING OFFICE 
REPORT TO THE HONORABLE PAUL S 
UNITED STATES SENATE 



PELL GRANT VALIDATION 
IMPOSES SOME COSTS AND DOES 
NOT GREATLY REDUCE AWARD 
ERRORS: NEW STRATEGIES ARE 
NEEDED 



D I^ G E S T 

The Pell program, administered by the Office of 
Student Financial Assistance in the U»S. Depart- 
ment of Education (the Department), receives more 
federal funds than any of the 5 other student fi- 
nancial aid programs funded under title IV of the 
Higher Education Act of 1965 and its amendments. 
The program awards grants for postsecondary educa- 
tion to students in financial need. In 1982-83, 
more than 2,5 million students received grants 
averaging $959 for total program awards of $2.4 
billion . 

Because studies of the 1980-81 program showed a 
large problem of inaccurate awards, the Department 
expanded the number of applicants required to doc- 
ument, or validate, their eligibility. More than 
1.6 million applicants were asked to verify infor- 
mation in 1982-83, 5 times the number required the 
preceding year. Recognizing the effort this would 
entail, the Department reduced the number of appli- 
cation items to be verified from 6 to 2. Appli- 
cants selected for validation had to bring to a 
school the documents that would prove the accuracy 
of their information on adjusted gross income and 
federal income taxes paid . (pp. 9-11 ) 

O BJECTIVES, SCOPE, AND METHODOLOGY 

The Honorable Paul Simon, formerly Chairman of the 
Subcommittee on Postsecondary Education of the 
House Committee on Education and Labor, asked GAO 
to examine validation's costs and burdens on 
schools and students, the origins and goals of 
the Department's methodology, and alternative 
approaches . (pp» 93-95 ) 

GAO employed several evaluation strategies. To 
answer questions on costs and burdens of 1982-83 
validation on schools, GAO used two "before and 
after" designs . (l ) To investigate costs in 
detail , GAO conducted in-depth case studies of 12 
postsecondary institutions representing the diver- 
sity of schools Pell recipients attend. (2) To 
obtain more generalizable information, GAO conduct- 
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ed a national mail survey of financial aid officers 
at a stratified random sample of 400 institutions 
participating in the program. (pp. 16-17) 

To investigate the effects of validation on stu- 
dents, GAO conducted a mail survey of students at 
3 different types of schools selected from the 12 
in the case studies. GAO compared the experiences 
of validated and nonvalidated students. Also, 
to obtain nationally representative data, GAO 
included questions on students* experiences with 
validation in its survey of financial aid 
officers . (pp. 17-18) 

To respond to questions on current validation 
policy and methodology, as well as alternative 
approaches to making more accurate awards, GAO 
reviewed the Department's most recent studies of 
Pell error and other documents and interviewed 
officials from the Department, the Office of 
Management and Budget (0MB), other federal pro- 
grams, national education associations, and the 
contractors that process grant applications and 
carry out error research. (pp. 18-19) 

"Error" in the Department's research and in this 
report refers to a wide variety of discrepancies. 
Labeling these "student" or "institutional" error 
does not always mean that applicants or school 
officials failed to act as they should have. The 
Department has no information from the Pell research 
on deliberate inaccuracy. (pp. 53-54, 58) 

GAO's findings are limited in several ways. (1) 
The ability to generalize from the student surveys 
is limited to the schools from which the samples 
were drawn. (2) Some cost data were based on re- 
call and may be subject to biases that GAO had no 
exact way of estimating. (3) GAO excluded some 
types of schools from the survey, so that its gen- 
eralizations are to a large proportion of, but 
not all, postsecondary schools in the Pell grant 
program. (pp. 19-20 ) 

VALIDATION ACTIVITY INCREASED 
SUBSTANTIALLY 

Validation increased substantially after 1981-82: 
institutions reported validating 39 percent of their 
Pell grant recipients in 1981-82, 64 percent in 
1982-83. GAO cannot determine how much of the 
increase came from Department requirements, since 
some validation was voluntary. In 1982-83, 81 per- 
cent of the schools reported validating more appli- 
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cants than the Department selected* Thirty-two 
percent also validated all their Pell applicants in 
1982-83, a 52-percent increase in the number of 
schools that did so in 1981-82 • The Department 
has never required 100-percent validation. (p. 23) 

Schools reported validating not only more stu- 
dents than the Department required but also more 
application items: 76 percent of the schools chose 
to validate items on dependency status, and 62 
percent validated other items such as nontaxable 
income. (p. 25 ) 

INSTITUTIONS EXPERIENCED SOME COSTS 
A ND PROBLEMS WITH VALIDATION 

Schools were generally positive toward validation 
and willing to see it expanded in some form to 
other financial aid programs, although they 
reported some costs and other problems with 
1982-83 validation. (pp. 34-37) 

From 1931-R2 to 1982-83, the schools increased 
their resources for validation, whether required 
or voluntary. On the average, they reported in- 
creasing staff time by one third. Both case study 
and survey data suggest Pell validation cost the 
schools an average of about $14 per case, but costs 
varied from $8 to $47 in the case study schools. 
At an average cost of $14 each, the 1.66 million 
validations the Department required in 1982-83 
cost schools an estimated $23 million. Many 
schools automated their data-processing and used 
other methods aimed at greater efficiency. Vali- 
dation costs v/ere higher for proprietary schools, 
schools with a constant influx of new applicants, 
and schools that handle comparatively few valida- 
tions. In some schools, there may be a limit to 
reducing cost by increasing efficiency. (pp. 
25-34, 120) 

Schools reported award delays (90 percent of the 
institutions ) and problems obtaining documents 
from government agencies (60 percent) and advising 
the parents of students undergoing validation (40 
percent). Payment delays, in particular, had 
negative consequences : they required schools to 
make special accommodations , such as deferring 
tuition, to help students over a delay. About one 
quarter of the schools with this problem had 
internal cash shortages when students could not 
pay their fees on time. The problems did not vary 
notably for different types of institutions, 
(pp. 34-35) 



VALIDATION DID NOT GENERALLY 
AFFECT STUDENTS NEGATIVELY 
BUT CAUSED PROBLEMS FOR SOME 

Although schools reported that students had 
difficulty obtaining documents from government 
agencies and with other steps of validation, 
validated students at the 3 schools surveyed 
generally did not report difficulty providing 
required information. Very few, regardless of 
their validation status, found it difficult to 
obtain Pell grants. (pp. 39-43) 

Validation problems and delays in awards seemed 
not to have a wide effect on most students' 
academic plans . Institutions reported that in 
1982-83, about 5 percent of the validated students 
changed their academic plans by enrolling late or 
deferring enrollment because of validation. GAO's 
findings suggest that some applicants — about 1 
percent — were deterred altogether from higher 
education by validation problems. Thus, schools 
estimated that about 69,000 students and potential 
students may have had their academic plans nega- 
tively affected in one way or another. (pp. 44-46) 

At the 3 case study schools, awards for about one 
fourth of all students were delayed. Validation 
was not the only cause of delay, but more valida- 
ted than nonvalidated students received delayed 
awards. Fifty-three to 84 percent of those who 
received delayed awards reported financial conse- 
quences. Most commonly, students borrowed money 
or made budget cuts until their awards arrived, 
(pp. 47-48) 

PROBLEMS WITH AWARD ACCURACY 
CONTINUED DESPITE VAI/IDATION 

The Department's research shows somewhat reduced 
error in 1982-83 but substantial, continuing 
problems with accuracy. In 1980-81, awards were 
inaccurate by at least $2 for 71 percent of the 
recipients, compared to 63 percent in 1982-83. 
The estimated total of all types of error in 
1982-83 was $649 million, equivalent ro about 27 
percent of the $2.4 billion awarded. (pp. 54-55) 

The two sources of error — student or application 
error and institutional error — are about equal in 
size. The Department's policy focuses on student 
error, but the 1982-83 increase in validation was 
not accompanied by an overall reduction in student 
error. 
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In 1980-81, student error was present in 38 per- 
cent of the awards. In 1982-83, following the ex- 
pansion of validation, the figure was 39 percent, 
including about 30 percent overawards and 9 per- 
cent underawards. Errors in adjusted gross income 
validated in 1982-83 declined about $22 million, 
but errors increased in other iteiTis, keeping over- 
all student error about the same. The estimated 
dollar value of student error totaled $328 
million. (pp. 54-58) 

In 1S^80-81, institutional error was present in 42 
percent of the awards* In 1982-83, the figure 
was 34 percent, including 16 percent overawards 
and 18 percent underawards. The decrease came 
mainly from finding fewer signed statements of 
educational purpose missing from school files, an 
improvement attributable to the Department's 
consolidation of forms. (pp. 59-61) 

Overawards were more frequent than underawards. 
In 1982-83, 42 percent of the students received 
overawards, 21 percent underawards. The average 
overaward ($444) was larger rhan the average 
underaward ($259). Overawards were more often the 
result of student error, underawards the result of 
school error. Thus, the consequence of the Depart- 
ment's stress on validation as -che corrective 
action was a closer focus on overawards than 
underawards. ( pp. 54-56 ) 

MODEST EFFORTS TO REDUCE ERROR 
WERE LIMITED BY INFORMATION GAPS, 
DECISIONS MADE UNDER PRESSURE, 
AND UNCOORDINATED MANAGEMENT 

The Department has attempted to reduce errors in 
Pell awards. It has studied the problem. It has 
expanded validation. It spent about $5.5 million 
between 1981-82 and 1982-83 on all aspects of 
validation at Department headquarters and the 
contractor that processes applications. And it 
has continued its institutional oversight, train- 
ing, and monitoring. GAO believes that this is a 
relatively modest effort, one with only marginal 
results in error reduction. (pp. 67-68, 77, 88) 

Decisionmaking on Pell error is made difficult 
by an absence of clear purpose and formal goals 
and targets for quality control. The Depart- 
ment has directed action at application items 
that result in overawards. It has done sig- 
nificantly less to underawards and school 
error, and it has done little to prevent 



rather than detect and correct error. (pp. 
67-68) 

The Department's reduction of error is hindered by 
many information gaps. The Department does not 
regularly monitor error rates at the institu- 
tional level. Data on error are based instead on 
occasional studies of national samples that varied 
in design, reducing the ability to compare the 
findings. The studies do not give reliable esti- 
mates of error rates at different types of schools, 
so that they cannot be targeted for correction. 
The studies have not evaluated the reasons for stu- 
dent or institutional error, limiting the ability 
to design corrective action. Information on pol- 
icy options is often limited, since the Department 
does not have a program of systematic tests of 
changes that might be made to increase both stu- 
dent and institutional accuracy. (pp. 62-64, 73) 

GAO found that decisionmaking on Pell error com- 
monly has time coristraints that make it difficult 
to consider complex data and weigh alternative 
strategies. Reacting late in 1981 to reports of 
error, the Department proposed a large increase in 
the number of validations for 1982-83 but with 
uncertain estimates of the cost of an initial 
proposal for central validation, no systematic 
data on the costs and burdens of the option final- 
ly chosen (expanding on-campus validation), and 
little time for considering alternatives. The 
rapid decisionmaking contributed to a misinterpre- 
tation of the data, the targeting of validation on 
only one of several sources of error, and inaccura- 
tely high forecasts of savings. (pp. 74-78) 

The Department's management problems hinder effec- 
tive action: the lack of policy on error and frag- 
mentation within the Office of Student Financial 
Assistance have led to confused responsibilities 
for identifying and acting on error. Specifical- 
ly, the responsibilities for identifying inaccura- 
cy and for developing and implementing strategies 
for correcting it are lodged in several differe.it 
offices. Results-oriented management is hampered 
because the Departmeiit does not plan or budget 
resources specifically for corrective action and 
does not track costs in relation to actual savings 
to judge efficiency. (pp. 75-77) 

GAO found the Department isolated from other zigen- 
cies tha'c deal with award inaccuracy, and the 
Department's officials do not examine strategies 
other agencies r.se. 0MB officials, concerned 
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about error in Pell awards, have pressed for 
action without specific suggestions or systematic 
efforts to identify alternative strategies that 
agencies might share. (pp. 80-81) 

MATTERS FOR CONSIDERATION 
BY THE CONGRESS 

Since the Department's administration of the Pell 
grant program lacks clear and coherent goals for 
award accuracy and well-defined strategies to 
achieve its goals, the Congress might consider 
whether additional guidance would be helpful to 
the Department. An error-free environment is 
clearly not possible, but §649 million in error 
seems excessive. 

Since GAO found many gaps in the Department's 
evaluative information on error, the Congress 
might consider whether the data that are available 
are sufficient for achieving accountability in the 
administration of the Pell grant program. (p. 89) 

MATTERS FOR CONSIDERATION 
BY THE DEPARTMENT 

Since the Department's piecemeal approach to aw^ard 
accuracy has problems of design, implementation, 
evaluation, and effectiveness, it might consider 
whether central weaknesses might be corrected with 
a comprehensive effort co improve its data, set 
goals for Pell grant accuracy, decide broad 
strategy^ and clarify internal management 
structure. (pp. 89-90) 

MATTERS FOR CONSIDERATION 
BY 0MB 

GAO is encouraged by 0MB *s requiring the 
Department to develop and implement a plan to 
reduce error in the Pell program, several of whos^ 
points are consistent with GAO * s analysis. GAO 
urges 0MB to continue to oversee the Pell program 
to insure that the issues GAO raises are considered 
and acted on. 0MB could assist the Department by 
informing it of practices elsewhere in the govern- 
ment that promise to promote accuracy. (p. 90) 

AGENCY COMMENTS AND GAO'S 
RESPONSE 

The Department and 0MB reviewed a draft of this 
report. The Department found no factual errors 
but disagreed with GAO's conclusions on manage- 
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ment. The Department described several corrective 
actions that were planned or under way after gao 
collected its data. While the Department believes 
that these form an overall strategy for reducing 
error, it also argues that only legislative change 
in design features of the Pell program will make 
error less likely. GAO believes that the initia- 
tives the Department described fail to address the 
error problem clearly and show a continuing lack 
of goals, an uncertain rationale for its choices 
of strategy, and inadequate plans to improve its 
data. The Department could do much more within 
the constraints of current law than it has done, 
(pp. 133-41) 

0MB concurs with GAO ' s findings and believes them 
useful. OMB calls the Pell error rate unaccept- 
able and has agreed to inform the Department about 
practices other agencies have found useful in try- 
ing to correct the problem of error. (p. 141) 
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PELL GRANTS AND THE PROBLEM 

OF ACCURATE APPLICATION DATA 

The Pell grant program receives more federal funds than any 
of the five other student financial aid programs funded under 
title IV of the Higher Education Act of 1965 and its amendments 
(20 U.S.C. 1070a (1982)). Administered by the Office of Student 
Financial Assistance v/ithin the Office of Postsecondary Education 
of the U.S. Department of Education ("the Department"), Pell 
grants are available to all students meeting specific eligibility 
criteria, the amount of a grant being based on the student's fi- 
nancial need. The students who receive Pell grants are enrolled 
in a wide range of postsecondary institutions, including colleges 
and universities, community and junior colleges, hospital schools 
of nursing, and vocational, technical, and business schools. In 
the 1982-83 academic year, the program awarded grants ranging 
from $50 to $1,800, at an average of $959, to help 2,522,746 
individuals further their postsecondary education. 1 In 1982-83, 
the Pell grant program awarded grants totaling $2.4 billion. 

Because data from 1980-81 had shown the continued problem 
of inaccurate awards, the Department in 1982-83 increased the 
number of students required to present evidence in support of 
their statements on their applications for Pell grants. The 
Honorable Paul Simon, chairman in 1983 of the Subcommittee on 
postsecondary Education of the House Committee on Education and 
Labor, asked us to investigate the costs and burdens of "valida- 
tion," or the verification of data on grant applications, on 
schools and students. He asked also about the origins and goals 
of the Department's validation methodology and alternative vali- 
dation policies and procedures that might be considered* 

This chapter begins with a sketch of the conflicting objec- 
tives that underlie this system for reviewing applicants' data. 
Then several kinds of background for understanding our data and 
conclusions are presented, including a description of how the 
Pell grant program works and the evidence of error in making 
awards that prompted the Department to take the validation 
initiative that is the main focus of our review. The chapter 
concludes with an outline of the questions that guided our review 
and the study designs and data sources we used to answer them. 

MULTIPLE OBJECTIVES OF THE PROGRAM 
FORM THE CONTEXT 'for GAO'S REVIEW " 

Any program that bases awards to individuals on their need 
must face the decision of how intensively to seek accurate data 

11982-83 is the most recent year for which there was complete 
program information at the time of our survey. 
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Table 1 



Pell Program Objectives and the Possible Effects 
of an Applicant Data Review System 



Objective 



Encourage students to 
attend postsecondary 
schools 



Award funds fairly 



Effect 



Positive 

Reducing overawards would save 
funds, which could be used to 
increase benefits for all; 
correcting underawards would 
give recipients tlie full aid 
they are eligible for; both 
would broaden recipients* 
choice of schools 

Would reduce errors, which 
would target awards better to 
individual need 



Negative 

The complexity of the control 
system may affect education 
plans by delaying awards and 
deterring some eligible students 
from applying 



None 



Respect autonomy of 
postsecondary 
schools 

Safeguard individual 
privacy 



Would improve autonomy if 
schools design their own 
controls 

Verifying application data 
from only one source document 
(or a few) would minimize 
intrusion 



Would reduce autonomy of schools 
if controls were fully 
prescribed 

Required documents would enter 
institutional records systems 



Minimize administra- 
tive costs 



Central direction of controls 
would encourage efficient and 
effective roathods 



Would have costs at all levels 
( government, contractors , 
schools ) 
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on that need: the various sources of data differ in accuracy, 
and efforts to verify accuracy have costs. The Department 
historically has gathered information about need directly from 
the applic'-ats for Pell grants and their families, by asking 
them questions on a form, and has done only a limited amount of 
checking of the answers. When several studies suggested that 
this information was sometimes inaccurate, the Department in 
1978-79 initiated the method of checking the applicants' data 
that it calls "validation." 

Validation is done by the schools in which Pell grant 
applicants are planning to enroll. Small in scale at first, the 
validation system was expanded greatly in 1982-83. With its 
expansion came growing concern that it might now cause costs and 
burdens for schools and students that could outweigh the bene- 
fits of making more accurate awards and that could also affect 
other objectives of the program. 

Table 1 shows some of the objectives that seem reasonable 
for characterizing the Pell grant program and some of the possi- 
ble effects of a system of reviewing applicants' data such as 
validation. Some objectives could be positively affected and 
others negatively. Some effects could go either way, depending 
on how controls are implemented. 

We were asked to do this review because of several uncer- 
tainties. First, there were no data on costs, burdens, and 
other effects of increased validation on institutions and stu- 
dents. Second, the Congress was uncertain about the quality of 
the Department's data on the extent of error in granting 
awards. The Congress needs data on both errors and the effects 
of corrective actions such as validation in considering Pell 
grant program policy and in weighing the importance of different 
program objectives. 

HOW THE PROGRAM WORKS 

To be eligible for a Pell grant, an applicant must meet 
specific requirements of citizenship or residence in the United 
States, be enrolled at least half time in an eligible program in 
a school participating in the Pell grant program by formal 
agreement with the secretary of the Department of Education, and 
demonstrate financial need. On the premise that the student and 
the student's family have primary responsibility for meeting the 
expenses of postsecondary education, one or the other of two 
formulas known as the "family contribution schedule" is used for 
assessing the family's financial strength and for estimating the 
need for financial aid that remains after the family's contribu- 
tion to expenses. 

The formulas are developed annually by the Department and 
reviewed by the Congress, and they consider indicators of finan- 
cial strength such as family income and assets, household size. 
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and the number of family members enrolled in postsecondary edu- 
cation. The formula for applicants who are dependent on their 
families considers income, assets, and other factors for the 
parents as well as the student. The formula for self-support- 
ing, or independent, applicants considers a student's (and a 
spouse's) income and assets, among other factors. The applica- 
tion form requests information on the applicant's living situa- 
tion and finances for the preceding and coming years in order 
to deterraine the student's status as either dependent or 
independent. 

For each applicant, a student-aid index is calculated that 
represents the extent of need or eligibility for Pell grant 
funds that remains after the expected family contribution. The 
smaller the index, the greater the financial need. The index, 
the cost of education at the student's chosen school, and the 
student's enrollment status (full-time or part-time) determine 
the amount of the award. The award increases as the student aid 
index decreases. An applicant with an index of 0 may receive 
the maximum award, which for program year 1982-83 was limited to 
half the costs of attendance and could not exceed $1,800. 

Eligibility is calculated by the Pell grant application 
processor, a private firm under contract to the Department, The 
firm is expected to provide a variety of services: receive the 
initial applications and later corrections to applications, 
enter each application into a computer file, use the computer to 
check the consistency of the information on the application, 
calculate eligibility, print and mail a student-aid report that 
includes the calculated index of eligibility for aid, send 
information on eligible students to states and institutions, 
select applications for data verification (and add special 
instructions to the student-aid reports for those that are 
selected), respond to questions from applicants in letters and 
by telephone, and merge records on applicants with records on 
recipients for statistical purposes after the close of a program 
year. System Development Corporation in Santa Monica, Califor- 
nia, was the contractor in 1981-82, 1982-83, and 1983-84, the 
3 program years of our reviev/, and the contractor for 1984-85 
through 1986-87 is National Computer Systems in Iowa City, Iowa. 

To receive a grant under the Pell program, an applicant may 
supply the specific data needed for determining eligibility on 
any one of five different forms. One is the Department's 
"Application for Federal Student Aid," which is sent directly to 
the processor. The four other forms that the Department accepts 
are used by state grant programs and the private need-analysis 
services that some postsecondary institutions require. The set 
of processing paths other than the federal form by which 
applicants may enter the Pell grant program is called "multiple 
data entry,'* 

These other forms are more extensive than the Department's 
form and require data that are not used in calculating eligibil- 
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ity for Pell grants. An applicant filling out one of the four 
forms who wishes to be considered for a Pell grant checks a box 
on the fom, and the entity that receives that application sends 
specific data for the applicant by computer tape to the Pell 
grant processing firm, which continues with the eligibility cal- 
culation and other processing steps. In recent years, only 33 
to 37 percent of the applications have been submitted on the 
federal form. The majority of the applications have come from 
the other sources, chiefly the College Scholarship Service of 
the College Entrance Examination Board, which accounted for 44 
to 47 percent of all Pell grant applications in 1981-82 through 
1983-84. 

For each applicant, the processing firm creates a computer 
file and checks the application data for internal consistency 
and validity. For example, the reported payment of federal 
income tax may be compared to the expected payment, v/hich can be 
calculated from the reported adjusted gross income and the legal 
tax rates, to see if the two figures agree. The processing firm 
suspends some applications for inconsistent or missing data, 
sending a form to the applicant and asking for a review and cor- 
rection or reaffirmation of the original data before proceeding 
with the calculation of eligibility. If the application data 
pass the computer checks, the formula is used to calculate the 
index, which is then sent to the applicant on the student-aid 
report. 

An eligible applicant completes the process by presenting 
the report to the officials of the participating school, who re- 
view additional eligibility criteria, calculate the award, and 
disburse the funds. An applicant may submit an application for 
aid for the coming school year any time beginning on January 1, 
but the schools disburse awards only between July 1 of that year 
and June 30 of the following calendar year, except in special 
circumstances. 2 Aid does not automatically continue from one 
year to the next; students must reapply for each subsequent 
school year for which they need aid but are eligible for grants 
for the time required to complete a first bachelor's degree. 



2The calculation of eligibility is based chiefly on data about 
the student and family circumstances in the previous year; a 
few items are forecasts for the coming year (in which the Pell 
grant would be used if awarded). An applicant may submit 
corrected data about the prior year to the processor at any time 
up to a predetermined cutoff date, on either the applicant's 
initiative, in response to questioning by the processor after 
the initial computerized review of the data, or the school s, 
following its review. There is no requirement that forecast 
items be corrected if they do not turn out as predicted. If the 
family situation has changed significantly for the worse since 
the previous year, an alternative "special condition applica- 
tion" form allows the calculation of eligibility to be based on 
estimated data for the current year. 
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FIgurt I: Pt<l Grant DcNwry Systtm 



Stud«nt 
Appl}c«Uon 



» Student fills out Poll grant or 
multipl* data ontry 
application 



Eligibility 
Oatormlnation 



• Poll grant processor com- 
putes studont>aid indox, an 
indicator of student's need, 
and produces student*aid 
report 

• School determines if student 

(1) has sufficient need and 

(2) meets general eligibility 
criteria for federal aid 

• School validates application 
data of sotected students 



Sourca oapirtmcnt of Education Request for Proposals. "Patt Grant Application processino 
Sy^tam." RFP-d3-OtO. December 10. 1982 p 9 



Funds Oliburstmant 

to Institution 



Banafit 
Calculation 



' School determines Size of 
student's U Qrant based on 
student's student-aid index, 
cost of attendance, an^ 
enrollment status (half, three* 
quarter, or full-time) 



• Department establishes 
school's authorization level, 
notifies school 

• School draws funds from 
Department's payment 
system up to authorization 
level 

• School submits studant-a*<. 
reports and progress reports 
on utilization of funds. 
Department uses reports to 
adjust school's authorization 







Fundi DIsbursomant 
to Studant 



> School disburses ^ell grant 
to student at least twice a 
year 



Account 
Rtconclliatlon 



• Department sends school 
studant validation roster, a 
list of school's Pcil grant 
recipients 

• School verifies or corrects 
report, returns it to 
Depart Tient 

• School collects over- 
payments or rafers cases to 
Department 
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BEST copy AVAILABLE 



of some 11,000 postsecondary institutions in the United 
States, in recent years about 7,000 have participated each year 
in the Pell grant program, taking responsibility for its day- 
to-day administration, following the regulations and policies 
for determining students' eligibility on dimensions other than 
financial need, calculating awards, disbursing grant funds, 
enforcing academic progress standards, and calculating and mak- 
ing refunds. 

once a school and the Department have entered into an 
agreement on the general terms of participation in the Pell 
grant program, the Department periodically advances funds for 
anticipated awards. The school may pay a student directly by 
check or by crediting the student's account at the school. 
Figure 1 is an outline of the basic Pell grant processing system 
as we have described it so far. In any given year, some number 
of the schools covered by agreements may not be authorized 
funds. Of the 7,334 schools covered by agreements in 1982-83, 
for example, only 5,852 were authorized funds. 

A school that wants -O have its students participate in the 
program but is not able or willing to take on all the duties of 
day-to-day administration may request the Department to calcu- 
late and disburse awards directly to the students. In 1982-83, 
of the 5,852 schools with funds authorized for their students, 
816 elected this option; they made up what is called the alter- 
nate disbursement system" while the remaining 5,036 made up the 
"regular disbursement system." Students who receive Pell grants 
under either system may receive financial aid from other title 
IV programs such as the Guaranteed Student Loan and the College 
Work-Study programs, state and institutional grants and loans, 
and other sources. 

A school that participates in the program must be accredi- 
ted by the appropriate accrediting organization, must be audited 
by an independent public accountant every 2 years, and must have 
a state license as an educational institution that meets minimum 
standards. The Department's Office of Program Review within the 
Division of Certification and Program Review periodically 
reviews the stewardship of the federal funds by examining the 
administrative capabilities, program compliance, and accounting 
practices at the participating schools. 

The pell grant program has grown dramatically since it be- 
gan in 1973. By the peak program year of 1981-82, recipients 
had grown 15 times the 1973-74 number, from about 176,000 to 
about 2.7 million. Total funds awarded each year grew almost 
fiftyfold in the same period, from $48 million to about $2.3^ 
billion. (program funds increased slightly in 1982-83, the lat- 
est year for which there is complete information, but the number 
of recipients dropped to about 2.5 n.illion. ) During the decade 
from 1973 to 1983, the average grant more than tripled, increas- 
ing from about $270 to about $959. When the program began as 
the Basic Educational Opportunity Grant (or Basic Grant) program 
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m 1973, only full-time freshmen were eligible to receive 
grants, but in 1976 eligibility was expanded to include all 
undergraduates (anyone not already holding a bachelor's degree) 
enrolled at least half time. The number of applicants has in- 
ffSJ^f? tenfold, from 0.5 million in 1973-74 to 5.1 million in. 

ERRORS IN PELL GRANT AWARDS 
AND THE DEPARTMENT'S RESPONSE 

TO award the correct amount to an eligible student and in- 
sure that the conditions of the award are met throughout the 
period it covers requires that many separate activities be car- 
ried out accurately and in proper sequence, sometimes under time 
pressure. (1) Students (and their parents in most cases) must 
obtain an application form and instruction booklet, gather 
household financial records, and correctly complete the detailed 
application. (2) The processors of the various types of non- 
federal forms raust quickly extract specific information from a 
specific group of the applications and encode it correctly onto 
magnetic tape to be sent to the Pell grant processor for further 
processing. (3) The processor must be ready with staff and 
equipment in multiple shifts to handle the enormous volumes of 
applications and corrections in the peak months of February 
through June, entering data, processing it, and printing and 
mailing the student-aid reports as rapidly as possible so that 
students and their parents can plan education for the coming 
year with some idea of the Pell grant funds that could be avail- 
able. (4) Institutional financial aid officials must review 
each student's eligibility details and accurately calculate 
costs and awards, also with a seasonally heavy work load. (5) 
Later, other school officials must track students' progress 
throughout the school year and inform the financial aid offi- 
cials when changes in the students' educational programs require 
adjustments in their financial aid. 

Errors are possible at each step—applying, determining 
eligibility, calculating and disbursing awards, and monitoring 
educational programs. The Department does not systematically 
monitor the administration of Pell grants for timeliness, cost- 
effectiveness, accuracy, and client satisfaction. However, 
scattered data are gathered on single criteria by different of- 
fices. For example, the Department's inspector general reviews 
inaependent auditors' reports on institutions' financial integ- 
rity. The processor maintains internal management data on the 
time taken in processing applications. Department staff review 
compliance with all student-aid rules by visiting several 
hundred schools a year in a process called "program review." 
Although the accuracy of the awards is affected by the actions 
of the students, the processor, and the institutions, it is not 
systematically monitored. The Department conducts occasional 
research studies to estimate the extent of error in awards in a 
particular year. 
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Table 2 



Student and Institutional Error Found 
in Two Pell Studies^ 

Student Institutional Total (absolute) 

Cases Cases Cases Average 

Program Sample with $ error with $ error with error per Total 

Contractor year size error (million) error (million) error recipient (million) 

Macrosystems 1978-79 2,309 43% $177 16% $70 55% $219 $248 

AdTech 1980-81 4,304 41% $352 37% $211 69% $346 $563 

^All error figures are based on $2 tolerance and calculated without counting errors of miss- 
ing affidavit of educational purpose or financial-aid transcript, which tends to understate 
error. The 1978-79 estimates are based on less rigorous verification of application data 
than those in the 1980-81 study (tending to understate student error in the former) and on 
discrepancies between verified application data and expected disbursements (the 1980-81 
study examined actual disbursements). 



In 1981, a Department report on the 1980-81 Pell grant pro- 
gram showed that errors of all kinds totaled an estimated $563 
million. The report indicated that an award problem noted in 
the Department's 1978-79 study of Pell grant accuracy had con- 
tinued, as table 2 shows. The 1978-79 and 1980-81 studxes are 
not completely comparable because of differences in the program, 
in the years in which it was examined and differences in study 
methods. We did not verify the reported estimates, but they do 
indicate a continuing problem of error in the program and have 
been interpreted this way by Department officials. 

The two main sources of error are in the applications and 
in the institutions. Students may omit data or state them inac- 
curately on their applications. The institutions may make mis- 
takes in determining initial or continuing eligibility and in 
calculating and disbursing awards. Processing errors are also 
possible but have not been extensively examined. A small-scale 
review that was part of the 1980-81 error study suggested that 
keystroke error in entering data from application forms to the 
computer terminal was low. 

The Department has made diverse responses to the problem of 
error. It has tried to lessen institutional error primarily by 
training the schools' financial aid officials in th'a Depart- 
ment's rules and procedures and by making program reviews and 
audits. Error in the applications has been the subject of a 
wide variety of initiatives. Efforts to deal with it have 
included (l) improving the language and design of the federal 
student-aid application form and its instructions, (2) expandxng 
the computer review of application data and questioning applx-_ 
cants about missing or discrepant items, (3) matching the applx- 
cation data with computer files to check the accuracy of reported 
Social Security and veterans' benefits, and (4) verifying the 
application data by comparing them with documentary sources 
before making an award. 
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^ Tho Department refers to the fourth approach as "valida- 
tion. The Higher Education Act of 1965 requires a person who 
wants a grant to file an application containing the information 
and assurances the secretary of the Department finds necessary. 
The program s regulations authorize the secretary and the 
schools to request the applicants to provide documents necessary 
for verifying the accuracy of the application information, 
including copies of the students' or parents' federal income tax 
returns. Failure to provide the requested documentation may 
make the applicant ineligible. The act also provides for crimi- 
nal penalties for knowingly and willfully submitting false 
information or otherwise obtaining a grant by fraud. 

The Department has required verification from some appli- 
cants but not from all recipients. Beginning in 1978-79, the 
Department used the processing contractor to notij.y selected 
eligible individuals that further documentation would be 
required when the student presented the student-aid report for 
payment at a school. The Department then required the schools 
to verify specific items on the selected applications. The 
selection criteria and the application items to be reviewed 
changed from year to year through 1981-82, but the proportion of 
applications to be selected remained constant at 6 to 7 percent. 

In response to the research study that showed continuing 
application and institutional errors in Pell grant awards in 
1980-81, the Department decided to focus on application errors 
and to require verification from many more applicants. In 
1982-83, 1.66 million applicants were asked to provide documen- 
tary evidence to financial aid offices in support of the data on 
their original applications— more than five times the 319,000 
applications that had been selected the preceding year. Recog- 
nizing that this required expanded effort from the schools, the 
Department reduced the number of items to be verified from the 
SIX required in 1981-82— household size, number of household 
members in college, dependency status, adjusted gross income, 
federal income tax paid, and nontaxable income — to the two 
required in 1982-83— adjusted gross income and federal income 
tax paid. Verification of other items was required for some 
applicants in both years, depending on discrepancies and other 
questionable statements on the applications. In both 
years, dependent applicants whose parents had not filed a 
federal income tax return were required to provide a signed 
statement from the parents attesting to this and to the accuracy 
of the data on the application. 

_ The Department provides extensive instructions and a vali- 
dation handbook to the financial aid officials who must carry 
out the verification. The instructions include rules for decid- 
ing when a mistaken application item requires formal correction 
(in which case the student must return the student-aid report to 
the application-processing contractor, requesting a revised cal- 
culation of the eligibility index). A school may issue partial 
awards even if the verification and recalculation of eligibility 

30 



have not been completed, but the school is liable for funds that 
are awarded in excess of the student' s need as determined by the 
final verification. 

The Higher Education Act of 1965 as amended authorizes and 
requires payment to the participating institutions of an annual 
administrative allowance of $5 for each Pell grant recipient. 
This payment may be used for the costs of administering the Pell 
program, including any validations, as well as the costs of 
administering other federal, campus-based student aid. 

PREVIOUS GAP REVIEWS OF THE PELL 
GRi?\NT PROGRAM 

We have reviewed the Pell grant program several times, most 
recently when we focused on the administration of the program at 
proprietary schools. 3 In that review, we reported evidence of 
institutional errors in 1980-81 in recruiting practices, admis- 
sion requirements, award and refund computation and disburse- 
ment, and the monitoring of the academic progress of the grant 
recipients. We did not examine practices of the schools, such 
as the verification of application data, that might respond to 
the problem of student error, observations in that review con- 
cerning the Department's weak data on error rates of all kinds 
and on problems with its monitoring and review of the institu- 
tions are relevant to our review of overall quality control m 
the Pell grant program. 

In 1981, we examined a 1979-80 sample of all types of 
schools in the program and found weaknesses in the Department s 
standards and the institutions' practices of monitoring grant 
recipients' academic progress. We did not review application or 
award matters. 4 In two reviews in 1977 and 1979, we reported on 
conditions in the first 4 years of the program that have changed 
considerably since then, but we found problems with application 
data during both reviews and recommended verification m the 
earlier one.-^ 



3u.S. General Accounting Office, Many Proprietary Sc hools Do 
Not comply with Department of Education's Pell Grant program 
Requirements , GAO/HRD-84-17 ^Washington, D.C.; August 20, 
1984) . 

4u.S. General Accounting Office, Students Receiving F ederal Aid 

Are Not Making Satisfactory Academic Progress; Tougher 

Standards Are Needed , GAO/HRD-82-15 (Washington, D.C.: 
December 3, 1981). 

5u.S. General Accounting Office, Office of Education's Basic 
Grant Program Can Be improved , GAO/HRD-77-91 (Washington, D.C.! 
September 21, 1977), reporting on data from 1974-75, and 
inconsistencies in Awarding Financial Aid to Students Under 
Four Federal Programs, GAO/HRD-79-16 (Washington, D.C.: May 
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OBJECTIVES, SCOPE, AND MBTHODOLOGY 



As the congressional request letter (in appendix l) 
indicates, this review was prompted by testimony to the Congress 
that raised general concern about the expansion of validation 
for program year 1982-83. We were asked to evaluate several 
effects of the Department's validation policy and to answer 
several questions on three other topics. 

The purpose of validation is either to confirm the accuracy 
of student application data or to discover errors and correct 
the data to insure that each award is based on the most accurate 
estimate of need. Thus, an important outcome of validation to 
measure is the extent of c.,^^ard error. We were not asked to 
undertake original data collection for measuring error. The 
Deparlunent had initiated the third in its occasional studies of 
Pell grant error and the effects of validation on error, and it 
was to be conducted in 1982-83 as validation was being expanded. 
However, we were asked to review the quality of the Department's 
study data and their interpretation. Table 3 shows the topics, 
the questions, and the major data-collection strategies we used 
in this review. 

Design issues posed by the study questions 
on institutional and student effects 
of validation 

As shown in table 3, our review involved descriptive, 
normative, and cause-and-ef feet evaluation questions and we 
employed a variety of evaluation strategies to respond to them. 
The central questions in the review, however, concerned the 
effects of the 1982-83 validation requirements on institutions 
and students (topics 1 and 2 in the table). Our major design 
challenge was to measure these effects, link them clearly 
to validation, and rule out other possible causes for the 
observations. 

Our final design for responding to these questions included 
several different strategies. The purpose of using multiple 
strategies is to provide information that has both depth (that 
IS, IS fully accurate) and breadth (that is, is generalizable) , 
since usually no single approach is satisfactory on both 
grounds. For example, one effect of a larger number of valida- 
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11, T979), reporting on data from 1976-77. No pre-award verifi- 
cation procedure was in effect during either review. The first 
report made direct recommendations; the Department of Health, 
Education, and Welfare agreed with the recommendation to in- 
crease and strengthen actions to verify applicant information 
and published pertinent regulations on January 25, 1979. The 
second report noted continuing discrepancies in application data 
and supported verification not only in the Basic Grant program 
but also in the other federal student-aid programs. 
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Table 3 



Study Topics and Questions on Pell Validation with Question Type, 
Data-collection Strategy, and Page Reference 



Topic 



1. 



Effects on 
institutions 



Question 

a. What costs are in- 
curred by the diverse 
types of institutions 
of higher education 
in doing the valida- 
tions? 



Question type Data-collection strategy Pages 



Descriptive 



b. 



Are there other Cause and effect 

effects of validation 

on institutions? Are 

these more serious at 

particular kinds of 

institutions? 



Effects on 
students 



a. 



What are the effects 
on students selected 
for validation? 



Cause and effect 



3. Methodology 



b. Do some effects fall 
disproportionately 
on particular groups 
of students? 

a» Does the Department 
base policy on reli- 
able* data on error, 
and have the data 
been interpreted with 
appropriate methods? 

b. Are the methods for 
selecting students 
for validation sta- 
tistically sound? 
Are they suited to 
policy goals? 

c. Does the Department 
evaluate its method- 
ology and use the 
findings in regular 
improvement of its 
approach? 



Current policy a. What is the goal of 
the Department's 
current policy on 
validation? 

b. How did the Depart- 
ment decide on its 
goals and methods and 
with What consid- 
eration of burden? 

c. What does it cost 
the Department to 
carry out its policy 
and methods? 



Descriptive 



Normative 



Normative 



Descriptive 



Descriptive 



Descriptive 



Descriptive 



On-site intensive case 31-34 
studies at 12 schools, re- 
viewing records and gather- 
ing staff estimates of 
1982-83 time use; review 
of comparable data from 
1981-82 study 

National survey of 400 23-37 
schools on validation ex- 
perience from 1981-82 to 
the present 



Mail survey of samples of 38-49 
students from 3 case study 
schools (n » 800), both 
validated and not; national 
telephone survey of eligi- 
ble applicants who received 
no grants; opinions of 
school officials gathered 
from 400-school survey 

Same as 2a 38-49 



Review of contractor re- 
ports; interviews with 
Department, contractor, 
and CMB officials 



Interviews with Depart- 
ment staff and contractor 
staff who developed and 
applied the methods 



Same as 3b 



62-64 



115-18 



Interviews with Department 
officials; review of 
records 



Interviews with Department 
and 0MB officials 



Review of Department data 
on estimates of staff and 
other costs and contrac- 
tor's on-site data on work 
volume and cost, using 
estimating factors from 
special study of past 3 
years of student 
corrections 



72-74, 
110-19 



66-68 



75-80 



70, 

120-21 
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(Table 3 continued) 
Topic 



Alternative 
approaches 
and method- 
ologies 



Question 



a. What are various 
approaches to the 
problem of error? 



Are there experi- 
ences of other fed- 
eral agencies, or of 
the private sector, 
that offer useful 
suggestions on this 
problem? 

Ate there alterna- 
tive methods of pre- 
venting or correct- 
ing award errors 
that could offer a 
better balance of 
positive and nega- 
tive effects? 



Question typ e 
Descriptive 



Descriptive 



Normative 



Data-collection strategy 

Review of past GAO work 
and pertinent literatures- 
interviews with staff at 
Department and at other 
agencies 

Same as 5a 



Analysis of all study 
data, including opinions 
on policy alternatives 
from officials in 400- 
school survey 



122-32 



122-32 



122-32 



tions could be higher administrative costs for financial aid for 
postsecondary institutions in 1982-83 • Adequately estimating 
these costs, and the portion attributable to validation, 
requires detailed interviews with staff to find out how they 
•pent time (since educational institutions typically do not keep 
such records) and review of school financial records on other 
costs • Given the effort required and normal constraints on 
resources, it is realistic to do this only on a limited sample 
of schools. Other kinds of effect can be estimated from survey 
responses, which are more efficient to collect and so can be 
obtained from a larger sample. 

Our design for studying effects on institutions included 
both approaches. We conducted case studies of costs at 12 
schools, and we sent a national survey designed to address the 
other issues to a carefully selected sample of 400 schools. The 
case studies of costs provided data whose strengths of detail 
and accuracy were complemented by the strengths of generaliza • 
bility and extensiveness provided by the institutional survey 
data. 

To answer questions about the effect on institutions of the 
increase in validation work, we included a "before and after" 
feature in the case studies and the national survey. In the 
survey, we asked institutions about 1981-82, the year before the 
increase in required validations; 1982-83, the year of the 
increases; and 1983-84, the year after the increases (which was 
still in progress when we did our survey) . The 12 sites for our 
case studies of financial aid administrative costs in 1982-83 
included 9 schools studied by the National Commission on Stu- 
dent Financial Assistance the year before. By using the same 
categories, definitions, and data-gathering methods in our study, 
we obtained data comparable to that in the 1981-82 study. The 2 
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years of data permitted before-and-after analysis of the cost 
effect of expanded validation at 9 of our 12 schools.^ 

Similar concerns for accuracy and generalizability surfaced 
in our design of the study of the effects of validation on stu- 
dents, and again we used multiple strategies. First, we sought 
to achieve acceptable accuracy in answers to our questions about 
students' validation experiences by using a mail survey. Since 
surveys of students typically have low response rates, we 
required that the mail survey sample be drawn from an informa- 
tion source that would provide up-to-date telephone numbers for 
follow-up. The most generalizable results would have come from 
a national survey of Pell grant recipients, but no national 
listings of students could provide telephone numbers. Thus, we 
decided to survey students at 3 schools rather than at schools 
scattered around the nation. We chose the 3 schools from the 
12 that participated in our cost case studies, because our cost 
case studies provided us with in-depth information on the vali- 
dation procedures required of students at these 3 schools (and 
at the 9 others) and, further, the schools could provide us with 
up-to-date student records including full names, addresses, and 
telephone numbers at school and at home. Our design for the 
student mail surveys thus was chosen explicitly, acknowledging 
the compromise between the need to have a nationally representa- 
tive sample of students, the need to have in-depth information 
about the specific validation procedures students faced, and the 
need for intensive follow-up to insure enough responses from the 
group for analysis. 

Second, in a partial effort to collect nationally repre- 
sentative data on validation's effects on students, we included 
in the institutional survey several questions asking campus 
officials to describe and estimate aspects of students' 1982-83 
experiences with validation. We asked about parts of the vali- 
dation process that students found difficult and about specific 
effects of validation on students' academic plans. 

Third, in order to be as confident as possible in attribut- 
ing effects on students specifically to validation, we included 
comparison groups in our design for the student mail survey. 
The survey questions asked validated students about experiences 
such as delay in receiving grant funds and changes in academic 
plans. However, if these experiences had happened commonly to 
nonvalidated students also, we would not be able to attribute 
the effects to validation. Therefore, to permit comparisons of 
the experiences of validated and nonvalidated students, we asked 
nonvalidated students about delays and changes in academic 
plans. 



6we added 3 schools to the 1981-82 case study group to include 
some types of schools not studied earlier and thus increase the 
group's representativeness of the diversity of postsecondary 
institutions. 
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Fourth, to insure that the description of effects of vali- 
dation would be as broad as possible, we expanded the study 
design beyond a search for the effects of validation on students 
who enrolled in school and received Pell grants. The difficul- 
ties of validation might have been so severe that some appli- 
cants would not have followed through to enroll in postsecondary 
education. To explore this possibility, we conducted telephone 
interviews with a sample of applicants who had been found to 
have the maxiir.um eligibility for a Pell grant but had never 
received one. We also asked the institutions in our national 
survey whether or not potential students had been deterred from 
pursuing postsecondary education because of validation. 

Data sources 

As we discussed in the section above on design issues, in 
order to obtain data of the depth and breadth we needed on 
effects on institutions (topic 1 in table 3 ) , we designed our 
study of institutional effects with two complementary approaches. 
First, we gathered data for national estimates from a mail survey 
of financial aid officers at a stratified sample of 400 insti- 
tutions participating in the Pell grant program.' The survey 
included questions about verification policy, volume of work, 
the resources used to do that work, the problems that were 
encountered, and views on the Department's requirements and 
alternatives. 

The 400 schools in our survey were all in the regular dis- 
bursement system and had financial authorization for 1981-82 
(the latest year for which the Department had a complete 
machine-readable roster of institutions). Our sample design 
called for schools that would be representative of 5-year-plus 
universities, 4-year colleges, 2-year community colleges, and 
less-than-2-year schools as well as being representative of 
public, private, and proprietary forms of control. When we 
examined these categorizations for branch campuses, however, we 
found discrepancies between a school's name and its classifica- 
tion — several community colleges were listed as 5-year universi- 
ties, for example. Department officials told us that a branch 
campus might be classified the same as the central carepus that 
administered Pell funds for the branch, even though the two 
schools might be quite different. This could occur in a state 
where the state university receives Pell grant funds from the 
Department and distributes them for the state's community col- 
leges. Since our sampling required accurate classification of 
each school by type and form of control, and since the Department 
could not tell us the extent of this kind of classification 
confusion for the 806 branch campuses, we excluded all branch 
campuses from the sampling frame. We did, however, send our 
survey to a random sample of 100 branch campuses; an analysis of 
the data showed the responses to be little different from those 
given by schools in the main survey. Technical details of 
sampling, response rates, and data analysis for the main insti- 
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tutional survey and the supplemental branch survey (as well as 
for the case study schools) are discussed in appendix II. 

Second, we obtained detailed cost information from on-site 
visits to 12 schools that represented well the diversity of the 
institutions that Pell grant recipients attend. Our teams of 
investigators spent 7 to 10 staff days at each school to review 
records, interview officials, and gather directly from staff 
their estimates of the time they used for validation and other 
financial aid procedures. We used a listing of six financial 
aid functions in order to gather time estimates that would allow 
an analysis of shifts in effort from one function to another as 
the validation work increased from one year to the next. Our 
cost case studies were performed under contract by the account- 
ing firm Touche Ross, which had done very similar case studies 
for the National Commission on Student Financial Assistance at 9 
of the 12 schools in the 1981-82 academic year. 

To answer the question about the effect of validation on 
students, we used three complementary data-collection strate- 
gies, as we mentioned in the section above on design issues. 
First, in our national institutional survey, we asked questions 
about validation's effects on students in 1982-83. Second, we 
surveyed samples of recipients at 3 of the 12 case study 
schools. We chose the 3 schools for their dif f erences--a 5-year 
private university with a policy of lOO-percent validation, a 
4-year private college defined by the Department as a tradition- 
ally black institution, and a public community college with an 
enrollment of low-income, urban minorities. At each school, we 
drew samples of about 250 students who were enrolled in 1983-84 
and receiving Pell grants. At 2 schools, half of these students 
had been selected for validation and half had not. At the third 
school, all recipients had been validated. 

Our student survey questionnaire asked the students ques- 
tions about their experience obtaining Pell grants for 1983-84, 
including the providing of documents and dealing with the proc- 
essor to get corrected eligibility. We asked all the students 
about delays in payment and changes in educational plans so that 
we could analyze the effects of validation on the students who 
were selected for validation and those who were not. Even 
though much of the rest of our review focused on 1982-83, the 
year of the major expansion of validation, we asked students 
about 1983-84 in order to stimulate the nearest and most precise 
recollections, despite the possible loss of comparability 
because schools had changed policies since 1982-83 or for other 
reasons. 

Third, in order to find out whether validation had been so 
difficult as to discourage some people from enr*olling in post- 
secondary schools, we planned to conduct telephone interviews 
with individuals from a sample of 2,000 applicants who had maxi- 
mum eligibility for a Pell grant in 1982-83 but who had never 
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received one. We obtained this sample from the Department's 
merged files on applicants and recipients for 1982-83, which 
contained 2-year-old addresses. When we attempted telephone 
interviews with these individuals to discuss their reasons for 
not enrolling, or for not accepting the Pell grants for which 
they had been eligible, and the effect of validation on their 
change in plans, we were able to trace very few from these old 
addresses. For the 1,084 persons we did trace, we found about 
140 usable telephone numbers. We were able to complete only 42 
interviews, including interviews with 23 who recalled that they 
had been selected for validation and 16 who recalled that they 
had not been selected. (The remaining 3 did not remember if 
they had been selected or not.) 

To answer the three methodological questions (table 3, 
topic 3 ) , we reviewed the two most recent studies of error in 
the Pell grant program (performed under contract to the Depart- 
ment for 1980-81 and 1982-83 by Advanced Technology) , and we 
interviewed Department officials and staff at the current and 
previous application-processing contractors on methods for 
selecting applications for validation. To understand the issues 
in the Department's evaluation and improvement of its approach 
to student error, we interviewed contractors' staff who had pro- 
vided data for the Department's evaluation and improvement 
efforts, and we discussed the use of these data in interviews 
with Department officials. 

For the three descriptive questions on current Department 
policy (see table 3, topic 4), we reviewed manuals, handbooks, 
circulars, correspondence, testimony and other statements at 
congressional hearings, congressional committee reports on the 
Department's requests for special appropriations for Pell grant 
validation, and other public information. We interviewed offi- 
cials in many different parts of the Department's Office of 
Student Financial Assistance, which administers the Pell grant 
program, and in the Office of Management and Budget (OMB). 

To gather data on the costs of the current policy, we asked 
the Department to estimate various specific costs of performing 
validations for students attending schools in the alternate dis- 
bursement system and of overseeing the national validation ef- 
fort at schools in the regular disbursement system. We accepted 
the D'ipartment' s estimates of time staff spent on validation 
functions, staff salaries, and other cost elements, but we com- 
puted personnel costs from the time and salary data that the 
Department provided. Labor is the primary element of the 
Department's costs, and we were not told of any comprehensive 
system of records that could provide a oetter source of data on 
staff time and effort. 

We cou**^ not determine the costs of validation at the 
application-processing contractor from vouchers or other head- 
quarters documents, so we gathered data from the contractor's 
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offices i^n Santa Monica, California. The element of the 
contractor's work that varies most importantly with the extent 
of the validation requirements is the volume of corrections 
submitted after the applicants have filed their initial 
applications. We requested a special study of a sample of 
application files in each of 3 years, beginning with 1981-82, 
which allowed the contractor to estimate the percentaoes of 
corrections applicants submitted after their student-aid reports 
had been selected for validation. We made the assumption that 
these corrections were attributable entirely to validation. We 
applied the percentages to various work-load measures and 
combined these with cost data in order to estimate how much of 
the contractor's production costs could be attributed to 
validation. For staff and subcontractors whose work could be 
directly assigned to validation, we gathered time estimates from 
supervisors and other personnel. We requested summary data on 
other cost elements from the contractor's records, which we 
accepted without verification. 

To answer the questions on alternative approaches to the 
problem of error in the Pell grant program (table 3, topic 5), 
we reviewed literature, asked financial aid officials in our 
national survey for their views on alternative policies, in- 
terviewed officials in other federal programs, and discussed 
future policy directions with Department officials, officers of 
national education associations, and staff at 0MB. 

STRENGTHS AND LIMITATIONS 
OF OUR REVIEW 

'The data reported in the following chapters have many 
strengths. Some topics, such as institutional costs of valida- 
tion, have not been studied before, to our knowledge. With 
respect to dosign, this study includes such features as 
comparison groups of students and before-and-after data on 
institutions that allow more precise analyses of effects than 
previously reported. However, specific characteristics of some 
of the data limit the precision of some estimates and the 
ability to draw general conclusions from therr. Significant 
limitations include the following: 

— Data from the national survey of institutions can be 
generalized only to 3,912 of the 5,009 institutions 
from which we drew our sample. The generalizations, 
or projections, cannot include the non^esponding 
institutions, branch campuses (although we believe 
them to be similar to the schools we did survey), or 
schools in the alternate disbursement system. 

— Data from the surveys of students at the 3 schools in 
our cost case studies can be generalized only to the 
general group of recipients of aid at these schools 
( excluding nonrespondents ) . 
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— Data on staff time and related costs of validation at 
the 12 case study schools can indicate what a similar 
group of school cj might report but cannot be used by 
themselves to estimate costs nationally. Our estimate of 
the total cost of validation at institutions in 1982-83 
rests not only on the case study data but also on the 
national survey results. 

— Data on costs at the case study sites, at the Depart- 
ment's headquarters, and at the processing contractor 
are estimates based chiefly on efforts to remember 
how staff used their time; such estimates have an in- 
herent degree of error because of weak or distorted 
recall. There is no exact way to estimate the extent 
or direction of bias in the reports of incividual 
employees or their supervisors* 

THE ORGANIZATION OF THIS REPORT 

Because the main interest prompting the request for this 
review was in the effects of expanded validation on institutions 
and students, we address these two topics first, in chapter 2. 
In chapter 3, we present evidence from the Department's research 
showing the continuing problem of inaccurate Pell awards and the 
results of our evaluation of that research. In chapter 4, we 
analyze the origins and present operation of the Department's 
policies on error in Pell awards, using three concepts 2 the 
goals of corrective action, strategies used to reach the goals, 
and management activities to support the strategies. Chapter 
5 gives a summary of the evidence on the effects of validation, 
the continuing problem of inaccuracy, and difficulties with the 
Department's current approach to the issue. It concludes with 
matters for consideration by the Congress, the Department, and 
0MB. 
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CHAPTER 2 



EFFECTS OF VALIDATION REPORTED 



BY SCHOOLS AND STUDENTS 



Testimony to the Congress and comments to the Department by 
the National Association of Student Financial Aid Administrators 
(NASPAA) aoout the proposal co expand the validation of Peil 
grant applicants for 1982-83 included predictions of widespread 
effects on institutions and students. In testimony before the 
House Subcommittee on Postsecondary Education and in correspond- 
ence to government officials, NASFAA expressed the concern of 
its members about the proposed expansion. In its testimony 
before the Subcommittee, NASFAA said 

"The Department has . . . maintained that by limiting the 
verification to only two data elements it will not impose 
any significant burdens on institutions and v/ill, at the 
same time, achieve significant savings in the Pell grant 
program. Further, they suggest that these steps will not 
create hardships on students. . . . 

"We respectfully disagree with the Department's contentions 
and feel that such drastic measures are uncalled for at 
this time. Our fear is these procedures will create 
unnecessary hardships on the very students the program is 
primarily designed to serve [and on institutions] . 

This statement reflected two specific concerns of the 
financial aid community. The first concern was that the work 
required by the increase in the number of applicants to be 
validated would not be fully balanced by proposed cuts in the 
number of application items to be reviewed. It was thought that 
the net effect would be much more work for the schools* already 
busy financial aid staff. 

The second concern was that students especially v/ould suf- 
fer from the proposed changes, primarily for two reasons. First, 
the new procedures might delay the processing of applications. 
Processing delays might in turn delay awards, which could seri- 
ously disrupt students" plans. Second, the required documenta- 
tion of income (such as federal income tax forms) might be very 
difficult for many students to obtain, particularly students from 
the most needy families. There was concern t?iat award delays and 
documentation demands would be so burdensome as to deter many 
students from enrolling in school at all. 



Oversight on Current Status and Administration of Federal 
Student Assistance Programs , Hearing Before the Subcommittee on 
Postsecondary Education, Committee on Education and Labor, U.S. 
House of Representatives, 97th Cong., 2nd sess. (Washington, 
D.C.: U.S. Government Printing Office, 1982), p. 303. 
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Although similar concerns had been raised in comments on 
the first validation regulations in 1978, by 1982 data were 
still not adequate to settle many questions about how validation 
works. Hence, NASFAA's members feared that the Department's 
proposal l;o expand validation for 1982-83 would create 
substantial new burdens for the institutions and students 
participating in the Pell grant program. 

Exploring these predictions was one of the main purposes of 
our study. As we discussed in chapter 1, to obtain in-depth 
information about the administrative costs of financial aid and 
the effects of the Department's changes on these costs, we con- 
ducted case studies of the administrative costs of financial aid 
at 12 postsecondary institutions. Nine provided before-and- 
after information on institutional costs and burdens. To obtain 
nationally generalizable information, we conducted a national 
mail survey of institutions, asking them to report changes in 
institutional costs and burdens associated with validation in 
1981-82, 1982-83, and 1983-84. 

To estimate the effects on students, we conducted a mail 
survey of validated and unvalidated students at 2 schools and 
students at a third school that validates all aid applicants. 
To explore the possibility that validation deterred students 
from pursuing a Pell grant and, thus, enrolling in school^ we 
conducted telephone interviews with validated and unvalidated 
Pell grant applicants who had maximum eligibility but did not 
receive awards. 2 Using multiple methodologies to obtain both 
in-depth data and generalizable information, we sought 
comparative data in order to strengthen any attribution of 
problems to Pell validation and changes in the program. 



We found evidence that the Department's expanded 1982-83 
validation procedures have increased institutional work loads 
for all types of institutions (public and private colleges, 
universities, community colleges, and proprietary, or 
commercial, schools). Validation activity at institutions has 
increased substantially since 1981-82, and the institutions have 
increased the resources they use for validation. However, most 
institutions reported voluntarily validating more students than 
the Department requires, and most have positive attitudes about 
the need for validation. As for students, some experience 
delays in awards and similar problems, but most seem not to find 
validation particularly troublesome, and relatively few seem to 
encounter problems that affect their educational plans. 

-^All mail survey data reported in this chapter are based on the 
actual number of responses to survey items. Unless otherwise 
indicated, the figures are projections to (or, rather, estimates 
for) the appropriate population of schools or students. All 
populations have been adjusted to exclude survey nonrespond- 
ents. See appendix ii for details about our samples and 
populations. 
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Modest effects on institutions, positive institutional 
attitudes about the need for validation, and limited effects on 
students are not the whole story, however. The Department's 
1982-83 changes in validation procedures appear not to have 
created the serious side effects on schools and students that 
many feared, but validation does have some costs for schools and 
some effects on students, whether it is required by the Depart- 
ment or done voluntarily by the institutions. An evaluation of 
validation must consider these costs and effects and balance them 
against evidence of savings from increasing the accuracy of pell 
awards. 

INSTITUTIONS REPORTED INCREASED VALIDATIONS, 
THE USE OF MORE RESOURCES, AND SOME PROBLEMS 

Table 4 on page 24 shows that the 3,912 schools in our 
respondent population reported validating 1,220,807 applicants 
in 1982-83 compared to 779,720 in 1981-82.3 m other words, 
these institutions reported that they validated 64 percent of 
their Pell grant recipients in 1982-83 but only 39 percent in 
1981-82. This increase in validation activity is consistent with 
the Department's greatly increasing the number of applicants 
selected for validation in 1982-83. However, the institutions 
reported that they voluntarily chose to validate more applicants 
than the Department required, even in 1981-82. 

Table 4 shows further that 32 percent of the institutions 
reported that they validated 100 percent of their Pell applicants 
in 1982-83. This is a substantial (52-percent) increase over the 
portion doing so in 1981-82. The Department did not require 100- 
percent validation in 1981-82 or 1982-83. Rather, it required 
schools to validate only applicants selected by the processing 
contractor (following the Department's criteria), and it has 
never required institutions to validate all applicants. 4 a sub- 
stantial and increasing portion of the institutions chose none- 
theless to do so. 



^The 3,912 institutions enrolled 76 percent of all Pell recipi- 
ents in 1982-83 and constituted 74 percent of all central and 
independent campuses in the regular disbursement system. We 
excluded alternative disbursement system campuses from our 
survey because they do not do validations and we surveyed 
branch campuses in the regular disbursement system separately. 
Their responses did not differ substantially from those of the 
central and independent schools. See appendix II. 

^The processor selects applicants individually for validation, 
basing the selection on characteristics of the application data 
and without regard for the school an applicant may attend 
(which is generally unknown when the student applies for aid). 
Thus, the actual number of validations any school must do is 
not planned or formally decided by the Department. 
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Further, even the institutions that did not validate 100 
percent of their applicants reported that they validated more 
applicants than the Department required. Table 4 shows that 81 
percent of these institutions reported that they went beyond the 
Department's requirements in 1982-83 and validated addition<>l 
applicants whose applications appeared suspect. Twenty-two 
percent also reported that they validated Pell applicants who 
applied for other financial aid such as institutional scholar- 
ships and loans. The proportion of institutions choosing to 
extend validation ^o these two groups of applicants was also 
greater than in 1981-82. 



Table 4 

Schools' Pell Validation Activity in Program 
Years 1981-82, 1982-83, and 1983-84^ 



Recipients 

Total number 

Average number per school 

Validated applicants 
Total number 

Average number per school 

Schools with 100-percent 

validation 
Schools without 100-percent 
validation 

Validated additional appli- 
cants with suspect data 
Validated additional appli- 
cants who also applied for 
other financial aid 
Schools validating dependency 
data 

Schools validating additional 
data (e.g., nontaxable 
income ) 



1981-82 1982-83 1983-84^ 

2,009,925 1 , 92 1 , 3 59 1,607,502 

524 501 424 

779,720 1,220,807 967,524 

208 318 252 

21% 32% 32% 

75 81 79 

16 22 29 

79 76 82 

57 62 67 



^Data are from our national survey and are estimates for our 
respondent population of 3,912 central and independent schools 
in the regular disbursement system developed by attaching weights 
to survey responses. 

lumbers (and possibly percentages) cannot be compared with 
earlier years in that they represent interim data on the 1983-84 
program year in progress during our survey. Schools reported 
validations from July 1, 1983, through October 30, 1983, and 
general intentions concerning validation policy for the whole 
year. 
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Finally, the institutions reported that they voluntarily 
chose to validate additional application items. In 1982-83, the 
Department required validation of only adjusted gross income and 
federal income taxes paid. However, 76 percent of the institu- 
tions validated application items used to decide dependency 
status, and 62 percent validated still other items such as non- 
taxable income. 

The 1983-84 program year had not ended when we surveyed the 
institutions, but reports of 1983-84 validation activities up to 
the time of the survey suggest that institutions intend to 
continue to go beyond the Department's requirements. The 
portion of institutions validating 100 percent of applicants, 
extending validation to applicants whose applications appear 
suspect or who apply for other financial aid programs, and 
validating additional data items all continued to be above 
1981-82 levels. 

Resources used for validation 
increased 

The institutions reported that they increased tue resources 
they used for validation as validation activity increased, 
whether required or voluntary. This is not surprising, for more 
work usually means more resource demands. Validation has 
financial costs, and these costs have increased since 1981-82. 
However, on the whole, our national survey of institutions and 
our 12 case studies of administrative costs for financial aid 
found that the increase seems modest compared to the increases 
in validation activity. 

Institutions made some staffing 
increases and adjustments 

The chief resource required for validation is personnel. 
To do validation, financial aid staff compare data on various 
documents such as parents' or students' federal income tax 
returns with information on the students' applications. Since 
each application is unique, the activity is labor intensive, 
especially when there are hundreds or thousands of applicants. 
Staff may also have other tasks and responsibilities associated 
with validation^ such as communicating with students to en- 
courage them to begin the application process well in advance 
of the expected payment date and advising and counseling 
applicants undergoing validation. Hence, it is reasonable to 
expect staffing increases and adjustments as validation activity 
increases. For instance, institutions might hire more staff or 
reassign tasks among staff who have to handle increased work 
loads. 

Our national survey found that some institutions made 
special staffing adjustments specifically in order to complete 
1982-83 Pell validations, whether required or voluntary. For 
instance, 42 percent of the institutions reported that they 
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Table 5 



Schools That Made Staffing Adjustments 
to Complete 1982-83 Pell Validations^ 



Adjustment" 

Required staff to work 

unpaid overtime 
Dropped or deferred other 

staff functions 
Hired additional staff 
Paid staff overtime 



Schools made adjustment 

To a great 
or very great 



To any extent extent 

42% 13% 

39 10 

12 2 
9 1 



^Data are from our national survey and are estimates for our 
respondent population of 3,912 central and independent 
schools in the regular disbursement system developed by 
attaching weights to survey responses, 

%ot exclusive categories; an institution may have made more 
than one adjustment. 



required staff to work unpaid overtime, and 39 percent reported 
that they dropped or deferred other work in order to complete 
1982-83 Pell validations, reducing staff contact with students, 
reducing other administrative or management functions, and re- 
arranging or rescheduling work loads (see table 5). 

However, not all the institutions reported making special 
staffing adjustments, particularly to a great or very great 
extent. For instance, 58 percent of the institutions did not 
require staff to work unpaid overtime at all, and 87 percent did 
not require staff to work unpaid overtime to a great or very 
great extent. ^ The same pattern holds for each of the other 
types of special staffing adjustments shown in table 5: drop- 
ping or deferring other staff functions, hiring additional 
staff, and paying overtime. 

Although many institutions did not make special staffing 
adjustments to a great or very great extent, on the average they 
used more staff time for 1982-83 validation, whether required or 
voluntary. The institutions reported on the average that they 
increased the time of professional staff 33 percent, student 
clerks 50 percent, and other staff 17 percent. In other words, 
across all types of staff, institutions increased the time staff 
spent on Pell validation by one third. Thus, in 1981-82, on 
the average, institutions spent 3.6 staff months solely on Pell 
validation, whereas in 1982-83, on the average, they spent 4.8 
staff months. Staffing seems to have held steady for 1983-84. 
See table 6. 
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Table 6 



Average Number of Full-Time Equivalent 
Staff Months Spent on Pell Validation 
in Program Years 1981-82/ 1982-83, 
and 1983-"M^ 

Increase 

Average number of months 1981-82 to 

Staff 1981-82 1982-83 1983-84 1982-83 

Professional 2.4 3.2 3.0 33% 

Student clerks 0.6 0.9 0.9 50 

Other 0^ 0^ 0/7 17 

Total 3.6 4.8 4.6 33 

^Data are from our national survey and are estimates for 
our respondent population of 3,912 central and independent 
schools in the regular disbursement system developed by 
attaching weights to survey responses. 



The average 33-percent increase in staff needed for vali- 
dation is attributable to large changes in a few institutions. 
Most institutions made no changes : 74 percent made no change 
for professional staff, 93 percent no change for student clerks, 
and 94 percent no change in other staff time. 

V7e were able to examine staff costs as well as other re- 
source costs in greater depth at our case study schools. 5 
these 12 schools, we examined in detail the 1982-83 costs for 
all administration activities for all financial aid programs, 
including the validation of Pell grants. For 9 of the schools, 
we compared 1982-83 costs with 1981-82 costs. 

Looking first at staff months as a measure of staff time 
and effort, we found that the 12 schools spent somewhat more 
time on 1982-83 validation than reported by the survey popula- 
tion. The case study schools varied greatly in the number of 
the staff involved in the full range of administrative tasks for 
all financial aid programs, not just Pell grants. Staff numbers 
ranged from 3.0 to 107.4 full-time equivalents in 1982-83. If 
we assume that a full-time equivalent staff year equals 12 
full-time equivalent staff months, our 12 schools had from 36 to 



^Case study data cannot be projected by themselves to the popula- 
tion of schools; all cost and resource figures from the 12 case 
studies are not weighted and apply only to these particular 
cases. Our national estimate of schools* validation costs is an 
approximation rather than a projection and rests not only on 
the case study data but also on national survey results. 
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1,289 staff months available for all administrative financial 
aid tasks. The time staff spent specifically on Pell validation 
in 1982-83 ranged from 1.2 to 22.8 full-time equivalent staff 
months, averaging 7.9. (The median and the most frequently re- 
ported number was 6.0 full-time equivalent staff months.) This 
is consistent with our survey findings^ 

At the 9 case study schools for which we had comparative 
data, the percentage of time staff spent on Pell validation 
increased somewhat between 1981-82 and 1982-83, going from 3 to 
4 percent. This is a 33-percent increase, also consistent with 
the survey report of a 33-percent increase. Like the survey, 
this overall increase conceals much variation in that only 4 
of the 9 schools increased time staff spent on Pell validations, 
and the quality of the data from 1 of the 4 schools is question- 
able. Of the remaining 5 schools, 4 showed no change in staff 
time in 1982-83 and 1 showed less time spent on pell validations. 

If we compare the number of validations performed in 
1982-83 with the number the preceding year, 8 of the 9 case 
study schools reported a larger number^of validations. If we 
exclude the school with questionable data, we fiml 7 of 8 
institutions reporting validation increases. The institution 
that did not report increased validations had already reported 
100-percent validation in 1981-82 and an increase in the number 
of items validated in 1982-83. 

We examined the average number of total staff hours (paid 
and unpaid) that the case study schools required in order to 
complete a single Pell validation in 1982-83. Setting aside the 
school that far exceeded the Department's validation requirements 
in both years and the school with questionable staffing data, 
we found that the remaining 10 schools used, on the average, 
from 0.5 to 3.7 hours of staff time to complete a validation, 
averaging about 1.7 hours. The 3 schools that spent the highest 
average effort were all proprietary vocational schools. 

In addition to looking at the actual amount of time (staff 
months and hours) that staff at the case study schools spent on 
Pell validation, we looked at the proportion of time staff 
spent on this task in comparison with all other financial aid 
tasks for all financial aid programs. As table 7 shows, staff at 
the 12 schools spent proportionally little time (13 percent) in 
1982-83 on validation and all other need-analysis and eligi- 
bility-determination activities for all financial aid programs. 
Staff spent proportionally even less time (4 percent) on Pell 
grant validation alone. Further, at the 9 schools for which we 
had comparative data across the years. Pell validation was a 
relatively minor function in 1982-83, even though the proportion 
of time that staff spent on it increased from 3 to 4 percent 
(and on the average involved 7.9 staff months in 1982-83). 

In sum, our national survey of institutions and our 12 case 
studies of financial aid administrative costs found that insti- 
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Table 7 

Average Percent of Staff Time Spent 
on Financial-Aid Administrative 
Functions in Program Years 
1981-82 and 1982-83^ 



Sample 



12 schools 9 schools 



Function 1982-83 1981-82 1982-83 

Outreach, counseling, and 30% 22% 27% 

receipt* and distribution of 
applications 

Accounting and collection 21 22 22 

Other (office planning, 13 17 14 

budgeting , personnel ) 
Packaging and awarding 15 17 18 

Need analysis, eligibility 13 12 12 

determination, and validation 
for all financial-aid pro- 
grams (including Pell) 

Validation of Pell grants 4 34 

alone 

Reporting and compliance 8 9 9 

^Data are from our case studies, are not population estimates, 
and may not add to 100 because of rounding. (Validation of 
Pell grants alone must be excluded when totaling columns. It 
is included in the overall function of need analysis and eli- 
gibility determination for all programs and shown separately 
as well on the indented line.) 



tutions made staffing increases and adjustments for 1982-83 
validation, whether required or voluntary. However, these 
increases and adjustments seem modest compared to the increases 
in validation activity. For example, our survey found that 
institutions increased the proportion of Pell recipients that 
were validated from 39 percent in 1981-82 to 64 percent in 
1982-83. Generally, they did this without making special staff- 
ing adjustments (such as requiring staff to work unpaid overtime) 
to a great or very great extent. On the average, they increased 
the time staff spent on Pell validation, whether voluntary or 
required, by one third. At the 9 case study schools for which 
we had comparative data, the increase in time staff devoted to 
validation also averaged one third. In both the survey and the 
case studies, however, some schools reported considerable in- 
crease, some no change, and a very few a decrease. Also, across 
all 9 schools in 1982-83, Pell grant validation continued to rank 
last among administrative functions for financial aid in terms of 
the proportion of staff time it required. 
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Institutions also made other 



resource adjustments 



Staffing change is not the only approach for coping with 
increased Pell validation activity. Schools can take a variety 
of steps to increase efficiency and productivity in handling 
validation as well as other Pell grant processing and management 
tasks. For instance, they can seek training and assistance for 
enhancing staff knowledge, and they can automate various 
application-processing tasks . 

Our national survey and our 1? case studies found that 
institutions have been using several strategies to handle 
increased Pell validation work loads. For instance, the 5 case 
study schools that did not increase or decrease the time staff 
spent on Pell validation in 1982-83 used more than one approach 
for handling the increased numbers of Pell validations. They 
increased automated data processing in various aspects of Pell 
grant administration (such as producing letters to students and 
tracking students through the stages of validation) and reallo- 
cated validation work from less efficient clerical or para- 
professional staff to more efficient professional staff, ^ 

Our national survey found also that institutions increased 
automated data processing for Pell validation and other Pell 
grant processing and management tasks, Twenty-four percent of 
the institutions reported that they increased the use of compu- 
ters specifically in order to complete 1982-83 Pell valida- 
tions. However, only 8 percent reported that they increased 
computer use to a great or very great extent. More generally, 
the institutions reported that they have steadily increased 
automation for several other aspects of Pell grant processing, 
which suggests that their general efficiency in administering 
Pell grants may be increasing. As table 8 shows, automation 
increased for all aspects of processing we asked about on our 
national survey. By 1983-84, 47 percent of the institutions 
reported that they were using computers for at least one func- 
tion (maintaining recipients' records). 

Institutions also sought and received special training and 
assistance for staff in the Department's 1982-83 Pell validation 
procedures. Sixty-two percent reported that they received for- 
mal workshop training from their state or regional professional 



^Staff time and dollar costs for automated data processing in 
the case study schools are included in the administrative costs 
for financial aid discussed in this chapter. 

■70ur survey did not ask specifically about reallocating valida- 
tion work from clerical to professional staff; however, it 
does show that more institutions increased professional staff 
time for validation in 1982-83 than increased clerical staff 
time. 
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Table 8 



Percent of Schools Using Automated 
Data Processing to Manage Pell Grants 
in Program Years 1981-82, 1982-83, 
and 1983-84^ 



Function 


1981-82 


1982-83 


1983-84 


Maintaining recipient records 


34% 


40% 


47% 


Maintaining applicant lists 


22 


30 


36 


Calculating eligibility 


11 


22 


25 


Calculating awards 


12 


20 


23 


Producing letters to applicants 


10 


13 


16 


Packaging multiple aid programs 


8 


12 


17 


Tracking documents for applicants 


6 


13 


16 



^Data are from our national survey and are estimates for 
our respondent population of 3,912 central and independent 
schools in the regular disbursement system developed by 
attaching weights to survey responses. 



associations, 61 percent from the Student Financial Aid Training 
Project operated for the Department by NASFAA. Sixty-two per- 
cent also reported that they made contact with the Department's 
regional or Washington, D.C., offices, either by phone or let- 
ter, to seek help with questions about validation requirements. 
Of the institutions receiving training or help from these 
sources, 84 to 88 percent judged it adequate or more than 
adequate; 1 to 5 percent judged it inadequate. The remainder 
were neutral. 

The dollar cost of validation varied 
but appears to have averaged about 
$14 per validation 

Our case studies suggest that the actual dollar costs of 
Pell validation for staff and all other resources and the 
1982-83 increases in "ihese costs varied considerably.^ Overall 
spending for Pell validation — whether required by the Department 
or done voluntarily — decreased at 2 of the 8 schools in this 



%Je excluded from all dollar cost calculations the school that 
supplied questionable data. Hence, we used 8 rather than 9 
schools for our analyses of cost increases in 1982-83 and 11 
schools rather than 12 for our analyses of the average cost per 
Pell validation in 1982-83. Further, for all comparative cost 
analyses, we adjusted 1982-83 dollars to remove the effects of 
inflation. We did not make this adjustment for our analyses of 
the average 1982-83 cost per validation, since no year-to-year 
comparisons were required. 
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Table 9 



Overall Spending for Pell Validation 
in 1982-83^ 

Cost increase 
1981-82 to 1982-83^ Cost 1982-83^ 
^22l2£l TotalQ proportion!! Total Average 

Proprietary 

Less-than-2-year $ 999 57% $ 3,219 $47 
business 

Less-than-2-year e e 12,435 42 
technical 

2-year business 1,117 30 4,984 27 



Private universityt 6,860 201 11,025 21 

Public 4-year college e e 16,558 17 

17 
15 
12 



State university 18,915 I14 37^192 

Conununity college -777 -7 n 575 

Private 4-year tradi- e e 12^534 

tionally black college 

State university 1 -4,068 -36 7,788 9 

State university 2 3,732 25 19,938 8 

Private 4-year college 5OO 18 3,464 8 

^Data are from our case studies and are not population estimates. 

DFor the comparison, 1982-83 costs deflated to 1981-82 dollars. 
1982-83 costs not deflated for analyses of costs per validation 
because year-to-year comparisons were not required. 

"Constant dollars. 

©Data on 1981-82 costs not available because not included in the 

study of 1981-82 financial-aid administrative costs. 
^Exceeds Department of Education validation requirements. 



analysis. At the 6 other schools, however, overall spending 
increased. See table 9. 

The dollar increases and the proportional increases in 
spending at these 6 schools varied widely, and no clear pattern 
of factors is associated with these increases. For example, not 
all the schools that increased staffing had cost increases. 
Conversely, some schools that did not increase staffing did have 
cost increases. The 2 schools with the largest dollar increases 
(and the largest proportional increases) were a state university 
that had to add staff in order to handle a much larger number of 
Pell validations and a private university that substantially 
exceeded the Department's requirements for validation in 1981-82 
and 1982-83. Total dollar increases at the 4 remaining schools 
ranged from $500 to $3,732. The median cost increase was $1,058. 

Looking next at the average cost per Pell validation, we 
found that (across the 11 schools in "this analysis) the overall 
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average cost per Pell validation was $14 in 1982-83. School- 
level average costs per Pell validation varied considerably, 
ranging from $8 to $47, as table 9 shows. The median school- 
level average cost per Peil validation was $17. The 4 schools 
;;ith the highest average costs, ranging from $21 to $47 per 
validation, were the 3 proprietary vocational schools and the 
school with a policy of doing much more intensive validation 
than the Department required. The schools with lower average 
costs per validation, from $8 to $17, included the larger 4-year 
colleges and universities.^ 

The. cost differences between these high-cost and low-cost 
schools appear to reflect partly differences in validation pro- 
cedures and partly differences in the size and characteristics 
of the student populations. For instance, the schools with 
lower average costs per validation generally had more extensive 
automation support for all administrative functions for 
financial aid, including validation. These institutions also 
performed many more validations than the high-cost schools. 
Hence, they may have achieved economies of scale and pi'ocessing 
efficiencies as staff became practiced in handling validations. 
Further, some por-^ion of the students being validated at the 
low-cost schools we.ve returning students who may have gained 
experience with financial aid procedures in earlier years. 

In contrast, the 3 proprietary schools tha^ had higher 
validation costs had short vocational programs of 6 months to 
2 years. (One of the 3 also offered 4-year vocational 
programs.) Thus, they had a smalle^ proportion of returning 
students (and a greater proportion of first-time applicants) in 
any group of aid applicants. These differences suggest that 
there may be limits to the cost savings that can be attained by 
means of processing efficiency. It may be that schools that 
handle comparatively few validations or that have a constant 
influx of new or first-time aid applicants will have higher 
costs per validation even when efficiencies are fully realized. 

Our national survey suggests that the average cost per 
validation of $14 in the case study schools may be a good 
working estimate of a general average cost per validation. Our 
survey asked, "In your opinion, what would be an appropriate 
administrative allowance (per student) to cover the costs of 
the Depart lent' s current Pell grant validation requirements?'' 
Responses ranged from $0 to $150, but 75 percent were estimates 
ranging from $0 to $15. The most common estimate, given by 
41 percent of the respondents, was $10. The average of all 
responses was $15. This figure is very close to our estimate of 



Our national survey data did not reveal notable differences in 
patterns of validation activity, resource use, attitudes 
toward or problems with validation, among types of schools. 
However, we did not examine the specific and detailed costs of 
Pell validation in the survey, as we did in our case studies. 
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$14 from the case study schools, although the question of an 
appropriate reimbursement is not identical to the question of 
actual cost. 

In sum, two major points emerged from our analyses of the 
resources institutions used for Pell validation, whether it was 
validation required by the Department or undertaken voluntarily* 
First, the institutions increased their resource allocations 
for validation. For example, in response to our survey, the 
institutions reported that they validated 64 percent of their 
Pell grant recipients in 1982-83 compared to 39 percent in 
1981-82. From 9 to 42 percent of the ins^:i\utions reported 
that they made some sort of special sraffAnr adjustment in order 
to complete 1982-83 validations, although ^. y from 1 to 13 
percent reported doing so to a great or very great extent. On 
the average, the institutions reported that they increased the 
time staff spent on validation by one third across all types of 
staff. They reported requiring 3.6 ataff months for validation 
in 1981-82 but 4.8 staff months in : 982-83. Our case study 
data also show increases but sugt^est that Pell :jr^:it validation 
ranked low aiaong all administrative functions for financial aid 
in terms of the proportion of staff time it required. Finally, 
both our survey and our case studies suggest that institutions 
made use of several approaches to increasing their general effi- 
ciency in administering Pell grants, such as increasing automated 
data processing for Pell grant raanagement tasks, including vali- 
dation, and obtaining training and assistance to increase staff 
knowledge. 

Second, validation, whether required or voluntary, cost on 
the average about $14 per validation. In short, the institu- 
tions invested staff time and other resources in conducting 
validation, regardless of whether it was required by the Depart- 
ment or done voluntarily as a matter of i'lstitutional policy. 
Further, higher costs at certain kinds of institutions with 
certain kinds of students suggest that there may be limits to 
the cost reductions that might be accomplished, no matter how 
efficiently the institutions carry out their own or the 
Department's validation procedures. 

In,;jtitutions had problems with validation 
but attitudes were generally positive 

The institutions reported that they had some problems as a 
result of, and with certain aspects of, the Pell gra it valida- 
tion process. However, on the whole, their attitudes toward 
validation seemed largely positive. One of the most con^aon 
problems for 1982-83 was the delay of Pell awaras^lO Ninety 
percent of the institutions in our national survey reported that 
payments to some Pell grant recipients were delayed in 1982-83 

lOln our survey, we did not ask about 1981-82 award dr. -.ays. 
Kence, we do not knov; whether delays increased in 19R2-83. 



specifically because of validation, whether required or 
voluntary. On the average, these institutions reported that 
awards to 17 percent of their recipients were delayed. 

These payment delays had two types o"^ consequence for 
institutions. Firsts 83 percent of the institutions whose 
students received delayed awards reported that they made special 
accommodations or adjustments in order to help the students over 
the delay. The institutions that chose to make accommodations 
reported that they most often deferred tuition or other fees 
until all Pell validation steps (including application cor- 
rections) were completed. They least often made partial Pell 
awards. Second, 26 percent of the institutions whose students 
received delayed awards reported that they experienced internal 
cash shortages when students could not pay tuition and fees on 
time. Although the 1983-84 award year had not ended at the time 
of our survey, this general pattern of delay and its consequences 
seemed to be holding. 

The institutions reported also that for 1982-83 they had 
particular problems with obtaining documentation of the receipt 
of Social Security, veterans' benefits, and welfare payments and 
counseling or advising the parents of applicants going through 
the validation process. Sixty percent of the institutions 
reported that obtaining the documentation was difficult or very 
difficult. Forty percent reported that advising parents was 
difficult or very difficult. 

The latter finding, that institutions had difficulty 
advising parents, is somewhat surprising, since validation is 
usually thought of as the students' responsibility. Our survey 
did not ask about specific Jifficulties in advising parents. 
However, several financial aid officers told us informally of 
difficulties that helped us interpret the survey finding. The 
anecdotes had a common theme: some parents tended to become 
puzzled and upset and to call the school when a student asked 
for sensitive financial documents such as federal income tax 
returns. This happened particularly among parents who were 
divorced or separated, were having financial difficulties, or 
had another child who had received a Pell grant without going 
through validation. 

Institutions' problems with Pell validation may decrease 
as time goes on. Sixteen percent reported that they found 
implementation harder in 1983-84 than in 1982-83. In contrast, 
35 percent of the ifxCtitutions reported that they found valida- 
tion easier to implement in 1983-84 than in 1982-83. 

Despite the problems, the general attitude of the insti- 
tutions toward validation appeared to be largely positive, 
although there was criticism. A positive attitude was exp^ ^osed 
both in institutions' practices and in their survey statements 
about various aspects of validation. Looking at their prac- 
tices, we found that the institutions voluntarily chose to 



Table 10 



Validation for Financial-Aid Programs 
Other Than Pell in Program Years 
^^981-82, 1982^3, and 1983-84^ 

1981-82 1982-83 1983-84 

Percent of schools valida- 48 58 60 

ting applicants for 
other programs 
Applicants other than Pell 

Total validated 693,711 1,012,515 1,144,321 

Average number valida- 181 263 297 

ted per school 

^Data are from our national survey and are estimates for 
our respondent population of 3,912 central and independent 
schools in the regular disbursement system developed by 
attaching weights to survey responses • 



Table 11 

Schools' Satisfaction and Dissatisfaction 
with Aspects of Department of Education Pell 
Validation Procedure's^ 

Neither Dissatisfied 
Very satisfied satisfied nor or very 
^SP^^^ or satisfied dissatisfied dissatisfied 

Clarity 

Application form 64% 23% 12% 

Validation hand- 61 26 14 

book 

Student-aid report 49 23 29 

instructions 

Timeliness in dis- 46 28 26 

tributing valida- 
tion handbook 

Consistency of rules 36 33 30 

year to year 

^Data are from our national survey and are estimates for our re- 
spondent population of 3,912 central and independent schools in 
the regular disbursement system developed by attaching weights 
to survey responses. 
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do more Pell validation work than had been required by the 
Department. In addition, as table 10 shows, the institutions 
reported that they validated applicants for financial aid 
programs other than Pell and that the number of these other 
validations increased in 1982-83. 

We found also that 69 to 87 percent of the institutions 
reported that they were either satisfied with or neutral about 
the aspects of the Pell validation procedures shovm in table 11. 
Twelve to 30 percent expressed dissatisfaction, particularly 
v/ith the clarity of the student-aid report and the consistency 
of the Department's validation rules from year to year.H 

In addition, 68 percent of the institutions reported that 
they supported, strongly supported, or were neutral about the 
proposal outlined in an August 1983 letter from the Department 
that would have extended some form of validation to other 
federal aid programs, including the Guaranteed Student Loan 
program. Specifically, 50 percent reported that they supported 
or strongly supported validating data from applicants for these 
other aid programs, 18 percent reported that they were neutral, 
and 32 percent reported that they opposed or strongly opposed 
the expansion. In contrast, only 25 percent of the institutions 
reported that they supported or strongly supported the Depart- 
ment 's proposed method for validating applicants for the pro- 
grams. The method would have X'equired that the institutions 
validate randomly chosen applicants, the number at each institu- 
tion to be based on its own error rates as determined from an 
annual study. Thirty-nine percent i-eported that they opposed or 
strongly opposed this specific method, and 29 percent reported 
that they were neutral about it. Seven percent reported that 
they were not familiar with the proposal. 

In sum, the institutions had problems with certain aspects 
of validation, whether required or voluntary. Particular prob- 
lems included delayed awards, difficulty obtaining documentation 
of Social Security, welfare, and veterans' benefits, and advising 
the parents of students undergoing validation. The institutions 
also criticized certain aspects of the validation procedures, 
particularly the 1982-83 student-aid report, for lack of clarity, 
and the Department's validation rules, for lack of consistency 
from year to year. In spite of these problems and criticisms, 
however, the institutions were generally satisfied with or 
neutral about most aspects of the validation process, and they 
seemed willing to see some form of validation expanded to other 
federal financial aid programs. 



l^After we conducted our survey, the Department substantially 
changed the format of the student-aid report. For 1984-85, it 
did not have a multipurpose single form but had three separate 
parts. We do not have information on institutions' reactions 
to the new format. 
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STUDENTS REPORTED SOME EFFECTS 



We explored the effect of validation on students from two 
different perspectives. First, in our national survey, we asked 
institutions for their estimates of effects on students in 
1982-83. Second, we asked four different groups of Pell 
applicants about their experiences with and reactions to the 
process of obtaining a Pell grant, including validation. One of 
the four groups consisted of applicants who had been found to 
have had great need for assistance and were eligible for an 
award m 1982-83 but had never actually received one. We 
surveyed these applicants in telephone interviews. The three 
other groups consisted of validated and unvalidated applicants 
who received Pell grants in 1983-84 and were students at 3 of 
our 12 case study schools. We surveyed these students by means 
of a mail questionnaire. 

We selected the schools for the student surveys primarily 
for the different types of students they served. One school is 
a public, urban community college with a high proportion of 
minority and low-income students and a higher proportion of 
independent students than the 2 other schools. Another school 
IS a private 4-year college most of whose students are and 
traditionally have been black. The third school is a private 
university with a national undergraduate and graduate student 
population. As a matter of policy, this school validated 100 
percent of its aid applicants. It also validated more Pell 
grant application items than the Department required. (See 
appendix n for more details of the student populations and 
samples at these schools.) 

^•^^ design for studying the effect of validation on students 

differed m two important ways from our design for studying its 
effect on institutions. First, in studying students we did not 
focus on the effects of the Department's 1982-83 changes, as we 
did for institutions. Analyzing changes requires either baseline 
data that have already been collected or accurate recall of ex- 
perience. We knew of no baseline data on the validation exper- 
iences of students, and it seemed highly unlikely that students 
would accurately remember the details of experiences that had 
occurred 2 or 3 years earlier. Hence, we chose to focus on 
students most recent experiences. 

Second, our applicant and student samples are not nationally 
representative, as is our institutional survey. Rather, the 
applicant sample was designed to provide a limited exploratory 
look at one particular group of 1982-83 applicants, and the three 
samples of 1983-84 students were designed to be representative 
only of the particular schools from which they were drawn. 12 

12Response rates and other aspects of the various data sets must 
also be taken into account. We discuss these in the chapter as 
we present the information from each data set. 
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Hence/ the comparison group design provides useful information 
on the effect of validation on students^ but our results can be 
generalized only to the particular schools from which the 
student samples were drawn. 

Most students seem not to have 
had problems with validation 

The institutions and the students reported that S9me 
students had problems with certain steps of validation. 
However/ they indicated that most of the students appeared not 
to have had problems or to have found it difficult to obtain a 
Pell grant. Further^ the problems appeared not to have fallen 
disproportionally on particular groups of students or to have 
been unmanageable. 

As table 12 shows, 22 to 62 percent of the institutions 
reported that students had difficulty with various aspects of 
the 1982-83 validation process. For example / 62 percent of the 
institutions reported that students had difficulty obtaining 
necessary documents from government agencies. Fifty- three 



Table 12 

Percent of Schools Rating Student Difficulty 
with 1982-83 Pell Validation Steps^ 

Very easy Neither easy Difficult or Don't 
Validation step or easy nor difficult very difficult know 

Obtaining documents 5% 32% 62% 1% 

from government 
agencies 

Understanding 13 34 53 0 

instructions and 
rules 

Obtaining timely 25 38 36 0 

and accurate 
corrections from 
processor 

Obtaining documents 19 51 28 1 

from parents 

Obtaining assistance 26 48 22 4 

and information 
from the Depart- 
ment and processor 

^Data are from our national survey and are estimates for our 
respondent population of 3/912 central and independent schools 
in the regular disbursement system developed by attaching weights 
to survey responses. 
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percent of the institutions said that students also had diffi- 
culties understanding the Department's rules and instructions 
for validation. 

Much of our data on students are limited to the 3 case 
study schools, but most of the students at these institutions 
who received pell grants in 1983-84 did not indicate that they 
had as many difficulties as the institutional reports suggest. 
Usually they reported that they had no difficulty providing the 
required information. At all 3 schools, most of the validated 
recipients reported that they had to provide their own or their 
parents' federal income tax returns. Across the 3 schools, only 
7 to 13 percent said that they found this difficult. Across the 
3 schools, 33 to 55 percent of the validated students reported 
also that they had to provide a signed statement from their 
parents to verify that some parts of their applications were 
correct. None of th6 community college students and less than 
10 percent of the students in the traditionally black college and 
in the private university said that this was difficult. 

Finally, across the 3 schools, from 0 to 20 percent of the 
validated students reported that they had to provide a variety 
of other documents such as official verification of Social 
Security, real estate appraisals, and income or benefit 
statements from state agencies. At the community college, 20 
percent of these who had to provide real estate appraisals said 
that this was difficult, while 2 percent or fewer of those who 
had to provide documents of the other types said that this was 
difficult. Similarly, at the 2 othr-r schools, 3 percent or 
fewer of those who had to provide veal estate appraisals or 
other documents reported that this was difficult. 

^differences between the institutions' reports of students' 
difficulties and students' own reports may stem from two main 
factors. First, our institutional survey did not ask institu- 
tions to estimate how many or what proportion of students had 
difficulty with the 1982-83 validation process, Protably not 
all students at a particular institution had difficulties. For 
instance, at the 62 percent of institutions reporting that 
students had difficulty obtaining necessary documents from 
government agencies, it may be that only a portion of the 
students had this problem. The student survey data from our 
3 case study schools suggest this interpretation. 

Second, it may be that 1983-84 applicants found validation 
easier than did 1982-83 applicants. As we discussed earlier, 35 
percent of the institutions reported that they found validation 
easier to implement in 1983-84 than in 1982-83, Further, our 
institutional survey and case study data suggest that the 
schools may have become more efficient in handling the manage- 
ment and processing tasks for Pell grants (for example, by 
greater automation). The students may have benefited if the 
institutions were more efficient with validation procedures. 
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The availability of assistance for students undergoing val- 
idation may also have helped make the process easier. Our 
institutional survey data and our case study data indicate that 
counseling students and giving them other kinds of help is a 
regular function of financial aid offices (see table 7). 
Validated students at all three schools reported that they 
received help from a variety of sources, most frequently the 
financial aid office at the school and the application 
processor. Across the three schools, 49 to 63 percent of the 
validated students reported that they received help from a 
school, and 20 to 26 percent reported that they received help 
from the application processor. At all three schools, most of 
these students reported that they were satisfied with or neutral 
about the help they received. Only 2 to 18 percent reported 
that they were dissatisfied. 

Several other findings from the three student surveys also 
support the general conclusion that most students do not find 
validation particularly difficult. For instance, some of the 
recipients at the three schools did not even recall having gone 
through the validation process. From 17 to 24 percent of the 
survey respondents whose records showed that they had been 
selected for validation did not recall having to provide 
documentation to verify information on their applications. In 
other words, they did not recall being validated. This suggests 
that for these students, at least, validation was neither a 
particularly important nor a particularly onerous requirement. 

It could be argued that the students' lack of awareness of 
the validation process is evidence that students 'cannot make 
valid assessments of the Pell award process. We believe that 
the students would have been much more likely to recall it if it 
had been onerous. In addition, most of our questions about 
validation were asked only of students who reported having been 
validated. Ignoring students who were validated but not aware 
of it presumably has the effect of inflating the students' 
reports of difficulty with validation. (This is because^ 
students who were validated but did not recall the experience 
were not asked how difficult validation was. Presumably, they 
would have reported minimal difficulty.) In brief, even the 
modest validation difficulties the students reported are 
probably somewhat inflated, given the exclusion of validated 
students who did not recall being validated. 

Finally, very few students, regardless of their validation 
status, reported that they found it difficult or very difficult 



13There is other evidence that students do not recall validation. 
The Department's study of errors in 1982-83 found that only 
about one fourth of the students who were sampled recalled 
having been selected for validation, whereas 60 percent had 
been selected. 
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Table 13 



Percent of Recipien ts Rating Diffic„i^y 
of Obtaining Pell Grants at Three SchonL 



in 1983-84^ 



Recipients 
All 

Validated 
Nonvalidatedb 



Public 
community 
college 

43% 

46 

37 



Easy or very easy 



Private 



4-year 
black 

41% 

39 

43 



University 



Difficult or very difficult 
Public Private 
community 4_year ~ 
college black University 



46% 

49 

33 



0% 

0 

0 



7 
1 



11% 
13 
7 



p^^^a^Ln^°? ?e^e^ f^^:^:^,-:^ r^^ll-J^^l our respondent 

P^-^^r^r^^- unfversity"^:;de^^?s^L1:Llefb? ^Sc^in^^^^ 

sciolPporicyf b^t'^p^rS^roflr^i"? — /o.be validated, according to 
provide'^docum;nts! and we cLssiSed'^hfrn""' ^'"t-^ "^""i"^ requirel to 

also placed studeAts in Jhe n™.,^^? nonvalidated" for this table. We 

in this category ?f they Sd nS? ?ecLri'^?^ traditionally black college 

sons within these schools are conJ^rv.i. '';"^ validated. Thus, compa?i- 

the "nonvalidated" studentJ mi oh^S ;''^ because some problems recalled by 
within the private u^ive^iitv^fiSr^ ""^^^ validation problems. Comparisons 
mation on recall effects ^' 1°0-P«^*=«"t validated school, provide infor- 
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to obtain a Pell grant. None of the conununity college students 
said that they found it difficult, as table 13 shows. At the 
traditionally black college, only 1 percent of the nonvalidated 
students and 7 percent of the validated students said that they 
found it difficult. The portion of students who reported dif- 
ficulty is higher at the private university, where 7 percent of 
the nonvalidated students and 13 percent of the validated stu- 
dents said that it was difficult or very difficult to obtain a 
pell grant. However, as a matter of institutional policy, the 
private university validated more application items than the two 
other schools. 



Table 14 



percent of Recipients Who Changed Application 
Data at Three Schools in 1983-84^ 



Recipients changing data 
All 

Validated 
Nonvalidated^ 



Public 
community 
college 

30% 

35 

21 



Private 



4-year 
black 

22% 

24 

18 



University 

43% 

44 

41 



^Data are from our survey of students and are estimates 
for our respondent population of 788 community college 
students, 848 students at a traditionally black college, 
and 248 private university students developed by attaching 
weights to survey responses. 

^All applicants at the private university were to be vali- 
dated, according to school policy, but a portion of the 
students denied having been required to provide documents, 
and we classified them "nonvalidated" for this table. We 
also placed students in the community college and tradi- 
tionally black college in this category if they did not 
recall having been validated. Thus, comparisons within 
these schools are conservative because some problems 
recalled by the "nonvalidated" students might have been 
validation problems. Comparisons within the private uni- 
versity, a 100-percent validated school, provide informa- 
tion on recall effects. 



Table 14 shows that validation was associated with 
application changes, even though it did not always cause 
students to make application changes and was not the only cause 
of application changes. At the public community college, for 
ex&mple, 35 percent of the validated recipients changed their 
applications, as contrasted with 21 percent of the nonvalidated 
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recipients. Both validated and nonvalidated students gave a 
variety of reasons for making application changes, ranging from 
correcting name spellings to correcting income information. The 
latter was the most common reason given by validated and 
nonvalidated students alike. 

Validation problems and delays in awards 
caused some students to make academic 
and financial changes ~ 

The question of whether or not validation affects students' 
academic plans is difficult to answer conclusively for several 
reasons. First, the question is tangled, is it that validation 
affects their plans or that it causes award delays, which in 
turn affect their plans? Second, validation difficulties and 
award delays do not necessarily discredit the validation 
process. Students and parents who are late in submitting and 
correcting applications may create validation problems more 
than the application processor or school officials do in 
processing and reviewing applications. Third, many personal 
factors influence students' academic decisions. Even at a 
moment of decision, it is very difficult for a student (or 
anyone else) to sort through all the factors and say which had 
the greatest influence. Fourth, it is impossible to gather 
absolutely reliable information on this complex topic from 
either the students or other informants. Students may not fully 
recall their reasons for making a particular academic decision, 
and the institutions probably never have complete information 
about why students make particular choices. 

With these complexities in mind, we examined validation's 
ecfects on students' academic plans in several ways. We asked 
-restitutions to estimate how much 1982-83 validation affected 
students' academic plans. We asked 1982-83 applicants who were 
greatly m need of and eligible for Pell awards but did not 
receive one whether or not problems with validation had been a 
factor m their failure to receive Pell funds. Finally, we 
asked the students at our three case study schools about the 
consequences of delays in their awards. Each of these data sets 
has Its limitations and must be interpreted cautiously. 

Overall, reports from the institutions and the 1982-83 
applicants suggest that validation problems did affect the 
academic plans of some students, causing some, for example, 
to delay enrollment until the next term or to change enrollment 
from full-time to part-time. These effects are reported for an 
estimated 60,616 students nationally, or about 5 percent of 
those who were validated. However, the plans of most students 
and applicants were not affected by problems with validation. 
Reports from the students at our three case study schools are 
consistent with the national survey and suggest that award 
delays (regardless of their cause) were consequential for some 
students, but most students were not affected. 
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Table 15 



Schools Reporting Student Changes 
in Academic Plans Because of Problems 
with pell Validation in 1982-83^ 



Change 

De layed enrol Iment 

To next term 

To later in the term 
Enrolled i'n another school 
Reduced full-time enrollment 
to part-time 



% of schools 



33 
20 
14 
14 



Total number 
of students 
making change 



23,984 

14;410 
11,234 
10,988 



^Data are from our national survey and are estimates for 
our respondent population of 3,912 central and independent 
schools in the regular disbursement system developed by 
attaching weights to survey responses. 



Table 15 shows that 14 to 33 percent of the institutions in 
our national survey reported that students made som« change in 
their academic plans in 1982-83 because of problems with 
validation. These institutions reported that students who were 
affected generally delayed their enrollment, for example, 
postponing enrollment until later in the term or until the next 
term. The number of students estimated to have made changes is 
modest compared to the total number of validated Pell grant 
recipients attending the institutions in 1982-83. For example, 
the institutions estimated that 23,984, or about 2 percent of 
the 1,220,807 recipients that they validated in 1982-83, 
postponed enrollment until the next term because of problems 
with validation. Similarly, the institutions estimated that 
about 1 percent of the 1982-83 recipients whom they validated 
delayed enrollment until later in the term and that less than 1 
percent changed from full-time to part-time enrollment or 
enrolled in another school. 

We did not attempt to evaluate the reasons for problems 
with validation. Therefore, whether a delay was caused by 
school officials, the processing contractor, the student, or 
the student's parents and others cannot be decisively 
concluded. Students' changing their academic plans because they 
had to take time to change incorrect information they initially 
submitted is, of course, their responsibility, not an "unfair" 
by-product of the validation process. 

We examined the question of whether students were 
potentially lost to postsecondary education because validation 
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deterred them. Our data suggest that most of the applicants 
were not deterred, although some might have been. For example, 
22 percent of the institutions in our national survey reported 
that potential students were lost to them in 1982-83 because of 
problems with validation, estimating the number at 8,526 
students. 1^ This is less than 1 percent of the 1,220,807 
validated Pell recipients enrolled by our population of 
institutions in 1982--83. 

Reports from our telephone interviews with 1982-83 
applicants who were greatly in need of aid and found eligible 
for but did not receive a Pell award also suggest that valida- 
tion problems may deter a few students but that most are not 
affected.!^ The 42 respondents gave a variety of reasons for 
not receiving the Pell grants for which they were eligible 
but did not generally include problems with validation among 
them. Twenty-two of the 42 said that they did not receive a 
Pell grant because they changed their minds about enrolling in 
school, and 21 said this was not because of validation but for 
family or personal reasons or because they preferred to work 
full-time. Only 1 of the 22 applicants linked the decision 
neither to accept a grant nor to enroll to difficulties with 
validation. 

Six of the remaining 20 respondents said that they enrolled 
in school but, discovering they did not need a Pell grant, did 
not take the award. They received other aid or used their own 
money. ^ This left 14 respondents, who gave other reasons for not 
receiving a Pell award. These included losing their applica- 
tions, enrolling for fewer than the minimum hours required, and 
having communication problems with the schools or the application 
processor, such as receiving a late notification of eligibility. 
Only 1 of the 14 did not receive an award because of difficulty 



'^As with all institutional reports about students, these find- 
ings must be viewed with caution, because institutions probably 
do not have fully accurate knowledge of students' actions, 
particularly students who did not enroll. For example, it may 
be that students decided to enroll in an institution without 
informing the institutions to which they had originally 
applied. similarly, it may be that students who did not enroll 
in 1982-83 were not lost to postsecondary education because 
they enrolled in 1983-84. 

^These findings also must be viewed with caution, for two 
reasons.^ First, our survey was designed to rovide exploratory 
information, not firm conclusions or numerical estimates, about 
the population in question. Second, we had great difficulty in 
locating individuals to interview, because the Department's 
mailing addresses for the applicants were 2 years old. We 
sampled 2,000 applicants, but after repeated attempts to locate 
1,084 of them yielded only 42 completed interviews, we 
terminated the survey. 
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with validation. In short, only 2 applicants of the 42 we inter- 
viewed linked problems with validation to not receiving a Pell 
grant in 1982-83 and, therefore, not enrolling in school. 

With regard to delay, students at the three case study 
schools reported that their wards were delayed, but as show 
in table 16, validation was not the only cause. For example, 8 
percent of the nonvalidated recipients at the community college 
and 13 percent at the traditionally black institution reported 
delayed awards. Validated and nonvalidated recipients listed a 
variety of reasons for delays, including the late submission of 
their applications, the time taken by schools to review the 
applications, ar.d the time taken by the application processor to 
make corrections. However, validation is clearly associated 
with the likelihood of delay. At the traditionally blacky 
institution, for example, 38 percent of the validated recipients 
but only 13 percent of nonvalidated recipients reported delayed 
awards. 

Award delays often affect academic plans. For instance, 
some of the recipients whose awards were delayed, regardless of 



Table 16 



Percent of Recipients Whose Pell Grants 
Were Delayed at Three Schools m 1983-84^ 



Recipients with delay 
All 

Validated 
Nonva 1 ida t ed'^ 



Public 
community 
college 

22% 
30 
8 



Private 



4-year 

black University 



29% 

38 

13 



33% 

38 

19 



^Data are from our survey of schools and are estimates 
for our respondent population of 788 community college 
students, 848 students at a traditionally black college, 
and 248 private university students developed by attaching 
weights to survey responses. 

'^All applicants at the private university were to be vali- 
dated, according to school policy, but a portion of the 
students denied having been required to provide documents, 
and we classified them "nonvalidated" for this table. We 
also placed students in the community college and tradi- 
tionally black college in this category if they did not 
recall having been validated. Thus, comparisons within 
these schools are conservative because some problems re- 
called by the "nonvalidated" students might have been vali- 
dation problems. Comparisons within the private univer- 
sity, a 100-percent validated school, provide information 
on recall effects. 
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Table 17 



Percent of Recipients with Academic 
and Financial Consequences frora Delayed 
Pell Grants m Three Schools in 1983-84^ 



Recipients with consequences 

All with grants delayed 
Changed academic course 

loads, schedules, etc. 
Borrowed money, maae budget 

cuts, etc. 



Public 
community 
college 

22% 
28 

53 



private 



4-year 
black 

29% 
8 

84 



University 

33% 
15 

66 



^Data are from our survey of students and are estimates for 
our respondent populations of 788 community college 
students, 848 students at a traditionally black college, 
and 248 private university students developed by attaching 
weights to survey responses. 



the reasons, reported that the consequences were negative. The 
consequences were more commonly financial, such as having to 
borrow money, than academic. Across all three schools, 8 to 
28 percent of the recipients whose awards were delayed reported 
that they experienced academic '::onsequences, but 53 to 84 per- 
cent reported that they experienced short-term financial con- 
sequences (see table 17). The academic problems reported were 
primarily having to change course loads or schedules or having 
to enroll part-time instead of full-time as planned. The 
financial problems were primarily having to borrow monev or 
to make budget outs until an award arrived. 

in summary, our national survey of inr. titutions suggests 
that in 1982-83, validation problems atrected some students' 
academic plans — for example, 5 percent of the recipients vali- 
dated by our population of institutions made some change in en- 
rollment status — but did not affect the plans of most. Roports 
from our student surveys and applicant interviews were consist- 
ent with the national survey. The student data suggest that 
being selected for validation wat^ ass )ciated with delays in 
receiving awards, although nonvalidated students also reported 
delays. About a third of the students who were validated 
reported delays, compared to about a tenth of the nonvalidated 
students. The data also suggest that awards delayed for what- 
ever reason were associated with both academic and financial 
adjustments for some students. About one sixth of the students 
who reported delays in receiving awards also reported academic 
changes such as enrolling in fewer courses, while about ^v/o 
thirds reported financial adjustments such as borrowing money. 
Most of the validated students seer, not to have experienced 
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either academic or financial consequences from delayed awards. A 
small proportion, however, may have had to delay enrollment or 
leave school, and some may have had short-term academic or 
financial problems. 

SUMMARY 

Testimony and correspondence from NASFAA to the Congress 
about a proposal to expand the validation of Pell grant 
applications for 1982-83 predicted effects more negative than 
positive. Two concerns were paramount. The first was that 
expansion would mean more work for already busy financial aid 
staff in postsecondary institutions. The second concern was 
that students would find it very difficult to provide required 
documentation and that their Pell grants would be delayed. There 
was even concern that delays and documentation demands would be 
so burder*some as to deter many students from enrolling in school 
at all. 

To explore these predictions, we sought to estimate the 
effects on the schools of the Department's changes in validation 
requirements from 1981-82 to 1982-83 and to describe the stu- 
dents' recent experiences with validation. Our data from stu- 
dents provide useful information about the effects of validation 
but are not nationally representative. 

We found evidence that the Department's expanded 1982-83 
validation procedures increased work loads for all types of 
institutions. For instance, institutions reported that they 
increased the proportion of Pell recipients that they validated 
from 39 percent in 1981-82 to 64 percent in 1982-83. However, 
some of this increase was voluntary: many institutions chose to 
go beyond the Department's requirements and validate more appli- 
cants and application items than it required. 

As validation activity increased, institutions allocated 
more resources for validation. On the average, they reported 
that they increased the time staff spent on validation by one 
third for all types of staff. Compared to the increase in 
validation activity, however, the resource increases seem 
modest. For instance, only 1 to 13 percent of the institutions 
reported that to complete 1982-83 Pell validations, they made 
some sort of special staffing adjustment (such as requiring 
staff to work unpaid overtime) to a great or very great extent. 
Pell grant validation still appears to rank low among all admin- 
istrative functions for financial aid in terms of the proportion 
of staff time it required. Institutions seem to have used sev- 
eral approaches for increasing their general efficiency in admin- 
istering Pell grants, such as increasing the automation of Pell 
grant processing and management tasks, including validation, and 
obtaining training and assistance to increase staff knowledge. 

We estimate from our case studies that the institutions' 
average cost of validating Pell grant applications was $14 per 
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validation. We noted in chapter 1 that the Department reimbursed 
the institutions $5 per recipient for all administrative costs, 
including validation. The pattern of higher validation costs in 
proprietary schools (possibly because of their short vocational 
programs and greater proportion of first-time applicants) sug- 
gests that there may be limits to the savings from efficiency in 
processing. 

The institutions seem to have positive attitudes about the 
need for validation, although there are some problems and some 
dissatisfaction with certain aspects of validation procedures. 
For excunple, institutions reported that awards are delayed and 
that they have problems in obtaining documentation of Social 
Security, veterans', and welfare benefits and in advi3ing the 
parents of students undergoing validation. Institutions also 
particularly criticized the 1982-83 student-aid report for lack 
of clarity and the Department's validation rules for inconsist- 
ency from year to year. In spite of these problems, however, the 
institutions report that they are generally satisfied with or 
neutral about most aspects of the validation process and that 
they are willing to see some form of validation expanded to other 
federal financial aid programs. 

Similarly, we found evidence that most validated students 
have not experienced difficulties with the process. The insti- 
tutions reported that some students do have difficulties with 
certain steps of validation, such as obtaining documents from 
government agencies and understanding the Department's instruc- 
tions for validation. Also, more validated than nonvalidated 
students in our three case study schools reported making changes 
in their Pell grant applications. However, students at these 
schools did not have as many difficulties as reports from the 
institutions might suggest. For instance, validated recipients 
at these three schools generally reported that they had no 
difficulty providing the required information. Further, very 
few recipients at these three schools (regardless of their 
validation status) reported that they found it difficult or very 
difficult to obtain a Pell grant. 

Validation problems and delays in awards seem to affect 
some students' academic plans, although the majority of the stu- 
dents seem not to be affected. For instance, institutions 
reported that, in 1982-83, about 1 to 2 percent of their valida- 
ted recipients made changes such as deferring enrollment to the 
next term because of problems with validation. About 5 percent 
in total may have had this type of experience. Our data also 
suggest that most applicants selected for validation are not 
deterred from higher education by validation problems but that 
some may be. 

With regard to other problems, our student surveys at the 
case study schools suggest that awards to validated students are 
more likely to be delayed: about one third of the validated 
students but about one tenth of the nonvalidated students 
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reported award delays. About one sixth of the students who 
reported award delays for any reason also reported changing 
academic plans (for example^ reducing the number of courses in 
which they enrolled) ^ and about two thirds reported having to 
make financial adjustments (such as borrowing money). 

Although our data are not nationally representative, most 
validated students seem not to make major changes in academic 
plans or experience other problems. With almost two million 
students being validated, a small percentage can, of course, 
involve several thousand students. However, we did not find 
evidence of effects as widespread and severe for students or 
institutions as might have been expected from the size of the 
validation effort required. 

Our understanding of the generally quite moderate costs and 
effects associated with validation should be placed in context. 
Validation is only one possible solution to the problem of award 
inaccuracies. It could be argued that the costs and burdens of 
validation are necessary and that the modest burdens we found 
are reasonable, especially if one assumes that the applicants 
made errors that ultimately led to many of the effects on 
students that we observed. A deeper question — whether vali- 
dation is the right remedy for the problem of error — involves 
issues such as prevention versus correction and alternative 
strategies for validation, which v/e explore in later chapters. 
In the next chapter, we assess the success of validation as 
implemented by the Department and the educational institutions 
in curbing Pell award errors. 
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CHAPTER 3 



INACCURATE AWARDS CONTINUE/ AND DATA 

ABOUT ERROR HAVE SHORTCOMINGS 

FOR GUIDING IMPROVEMENTS 

Given our finding that the present validation process does 
impose some costs and burdens, as described in chapter 2/ it is 
important to ask if these are balanced by reductions in award 
error. Finding that they are would not only help justify the 
costs of validation but would also suggest the appropriateness 
of validation as a way of reducing error. We used the Depart- 
ment's two most recent studies of Pell awards to compare the 
extent of errors in 1982-83/ the year of the major expansion of 
validation, with error rates in 1980-81. 

TYPES OF ERROR AND THEIR SOURCES 

The Department has no regular or routine process of 
sampling Pell awards and analyzing errors. The Department has, 
however, recently conducted two studies of Pell grants .1 Each 
study used a national sample of about 4,000 awards, from which 
the Department estimated the proportion of inaccurate Pell 
grants and the dollars awarded in error. Errors were identified 
by comparing reported data that were used in making awards with 
the "best value" for each of the same data items. These 
reported data were figures from the students and their parents 
that were used to calculate the eligibility index and figures on 
costs and enrollment status that were used by school officials 
to calculate the awards. 

The Department determined each "best value" from student 
and parent interviews, certified copies of federal income tax 
returns from the internal Revenue Service, bank and property tax 
records, and school records showing data used in calculations 
performed by school officials. The Department also looked to 
see whether the required documents were in each award file. 



^The Department has done three studies of the accuracy of Pell 
grants, in 1978-79, 1980-81, and 1982-83. Results from the first 
two are shown in table 2 in chapter 1, in this chapter, we do 
not discuss the earliest study because it is difficult to com- 
pare in detail to the two more recent studies, which reflect 
substantial changes in the Pell program after 1978 and differ 
significantly from the first study in methodology. See Quality 
in the Pell Grant Delivery System » vol. 1, Findings (Reston, 
Va. : Advanced Technology, 1984); Quality in the Basic Grant 
Delivery System , vol. 1, Findings (McLean, Va.: Advanced Tech- 
nology, 1982 ) ; and Michael T. Errecart et al . , Basic Educational 
Opportunity Grant Quality Control Study , vol. 1, Findings and 
Recommendations (Silver Spring, Md.; Macro Systems, 1979). 
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Comparing reported data with the "best value" showed the fre- 
quency of error for each item or group of items. To find the 
payment consequences of each error, the Department compared an 
award that was based on the reported data with what the student 
would have received if the error had been corrected by using the 
"best value." The Department attempted only to verify reported 
data; the study methods did not include searching for information 
omitted by applicants, such as undisclosed income or assets. 

Inaccurate Pell grants can be overawards and underawards. 
An overaward is an award that is either larger than it should be 
or granted to a student who is not eligible for any grant. An 
underaward is an award that is not as large as it should be or 
the denial* of a grant to a student who is eligible for it. The 
Department's studies of error did not try to find applicants who 
were eligible but through error received no award. Thus, the 
reported data on underawards include estimates only of the 
extent to which actual recipients received grants that were not 
as large as they should have been. 

To describe sources of error, the Department combined data 
on accuracy at every step of the Pell grant process into two 
broad categories called "student error" and "institutional 
error." Both can result in either underawards or overawards. 
"Student error" is application error. It includes, for example, 
the incorrect reporting of a student's status as a dependent and 
incorrect figures for income or household size. The overall 
amount of student error in an award is the difference between the 
award the student should have received if all the best values had 
been used at every step and the award calculated from the origi- 
nal application data and the correct institutional data. This 
procedure isolates the effect of incorrect application data. 

Institutional error ir.cludes a broad range of problems re- 
flecting the numerous tasks carried out by institutions after a 
student presents a student-aid report for payment. The compo- 
nents of institutional error shown in the Department's studies 
include an incorrect determination of the student's eligibility 
or enrollment status (full-time or part-time), an incorrect cal- 
culation or disbursement of the award; the use of incorrect fig- 
ures on the cost of attending the institution, and the absence 
of documents required by the Department's rules. (In cases in 
which documents were missing, the Department considered that the 
entire amount of an award was in error, although supplementary 
analyses disregarded the omission of documents and considered 
the individual eligible.) Institutional error in an award is 
the difference beteen a student's award and the amount the 
student should have received had the award been calculated 
from the eligibility index in the school'^ files and correct 
institutional data . 

For both student and institutional error, no information is 
available on whether inaccuracies were intentional (which could 
imply fraud or abuse) or unintentional. Unintentional error can 
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come from the misinterpretation of a question. Items requiring 
forecasts of family circumstances months in the future may also 
be particularly subject to unintentional error. Throughout this 
report, we use "error" and "inaccuracy" synonymously. 

We evaluated the Department's Pell error lesearch that had 
used the procedures described above. We found that the aggregate 
estimates of the incidence of error throughout the program, and 
the payment consequences of the various kinds of error, were 
useful, within the assumptions and study design limitations of 
the Department's work. (Detailed results of our evaluation of 
the Department's studies are presented Iciter in this chapter.) 
The Department acknowledges some reasons why its estimates could 
be incomplete or inaccurate. These include not only the lack of 
any direct estimate of errors of omission on the applications or 
errors of denial of eligibility, as noted above, but also the 
fact that the data were gathered while a program year was still 
in progress, before files and records were final, which might 
mean that some self-correction by institutions was not recog- 
nized. Thus, when the data are used, the possibility of some 
inaccuracy in the estimates should be recognized. Nonetheless, 
the data are adequate for examining the appropriateness of the 
validation process as a remedy for the types of error observed. 

FINDINGS FROM THE STUDIES 
OF PELL GRANT ERROR 

In the 1980-81 study, the Department found that 71 percent 
of the Pell grant recipients were awarded grants in error by at 
least $2 and that the estimated total of both underaward and 
overaward errors was $681 million. 2 Fifty percent of the reci- 
pients received overawards and 21 percent received underawards, 
although the dollar amount overawarded exceeded the amount 
underawarded by more than 4 to 1 . Student error was present in 
38 percent of the awards, institutional error in 42 percent. 3 

When the results of the 1980-81 study were reviewed in 
1981, the Department focused its action on student error, which 



^For these comparisons, we used the most stringent criteria: 
allowing a $2 error tolerance and considering an entire award as 
an overpayment if any required document was missing from the 
av/ard file. Thus, our error figures for 1980-81 are higher than 
those given in table 2. Later in the chapter, we report an 
analysis using the least stringent criteria: allowing a $100 
error tolerance and ignoring missing documents. 

^Since both student and institutional error could have 
occurred on any one application, the percentage of recipients 
with at least one error of either type is less than the sum 
of the percentage of recipients v;hose applications had at least 
one student error and the percentage of recipients whose appli-- 
cations had at least one institutional error. 
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Table 18 

Absolute Error in 1982-83 Pell Grants^ 



Programwide estimates 



Mean error/ 

recipient 
with error 



Source 



Error as 
Total error % of all $ 
(million) awarded^ 



Recipients 
with error 



Institutions 
Students 



$321 
328 



13% 
14 



33.5% 
39.4 



$379 
328 



Total 



$649 



27% 



62.7% 



$408 



^Data are from the Department of Education and include all 
overawards and underawards added together, without allowing 
them to offset each other within a single case or in the 
aggregate. Error is defined as a discrepancy of plus or minus 
$2 from the best award. 

^1982-83 Pell awards totaled $2.4 billion. 



chiefly resulted in overawards. (The 38 percent of awards with 
student error included 29 percent with overawards and 9 percent 
with underawards.) In a corrective action, the Department 
expanded validation for 1982-33 so that it would include many 
more students, although it required the verification of fewer 
application items by campus officials than in previous years 
(discussed in chapter 2). Institutional error, which included 
overwards to 26 percent of recipients and underawards to 16 
percent, appears to have been given no additional attention 
beyond the Department's usual attention to institutional 
compliance with program rules and the provision of voluntary 
training programs for school financial aid officials. 

The Department's study of 1982-83 Pell grants found a 
continuing problem of substantial orror, despite the increase in 
validation activity. The total absolute error (overawards plus 
underawards) was high in 1982-83, essentially showing only a 
small decrease from the error in 1980-81. In 1982-83 y the 
estimated total absolute error was $649 million. This amount is 
the equivalent of about 27 percent of the $2.4 billion awarded 
in Pell grants in 1982-83. 

On some indicators, there were some improvements from 1980- 
81 to 1982-83. Table 18 shows that the awards of an estimated 
63 percent of all recipients were in error by $2 or more in 
1982-83, down from the figure of 71 percent in 1980-81. The 
size of the average error per recipient also decreased between 
1980-81 and 1982-83. Although numerous technical issues make 
comparisons across the studies difficult to interpret, the 
comparison of percentages and averages (such as we have reported 
in this paragraph) may be more meaningful than other statistics. 
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THE SIZE AND DIRECTION OF INACCURATE AWARDS 



The Department's 1982-83 data show that an estimated 41.5 
percent of the recipients were given awards that were too large 
and that 21.2 percent were given too little money. Overawards, 
on the average, were larger ($444) than underawards ($259). 
Some of the overawards and underawards were substantial. For 
example, 14.1 percent of all the recipients, or an estimated 
360,000 students, received more than $550 more than they should 
have, while 3.8 percent, or 96,000 students, received more 
than $550 less than they should have. Counted among the 41.5 
percent who received overawards are the 29.8 percent of all 
recipients who were eligible for an award of some amount. 
However, 11.7 percent of all the recipients, or approximately 
300,000 students, should have received no award. More than half 
of all the funds that were overawarded went to students who were 
ineligible. According to the Department's estimates from its 
study results, 10 percent of the Pell program funds went to 
students who were ineligible for awards. 



STUDENT ERROR 



As shown in table 19, student error m 1982-83 resulted 
primarily in overawards. It was much the same in 1980-81. Only 
9 percent of the 1982-83 awards contained student error result- 
ing in underawards, while 31 percent of awards had student error 
leading to overawards. The total absolute student error was $328 
million. Overall, student error, which was present in 39 percent 
of the 1982-83 awards, persisted; in 1980-81, student error was 
present in 38 percent of the awards. 

The Department examined the extent of error in individual 
items and groups of items on the applications. As shown in 



Table 19 

Student Error in 1982-83 Pell Grants^ 
Programwide estimates 



Error 

Underawards 
Overawards 

Total 



Total error 
(million) 

-$ 55 
272. 

$328 



Error as 
% of all $ 
awarded^ 

2% 
11 

14% 



Recipients 
with error 

8.8% 
30.6 

39.4% 



Mean error/ 

recipient 
with error 

-§ 249 
351 

$ 328 



^Data are from the Department of Education. Error is defined as 
a discrepancy of plus or minus $2 from the best award. Totals 
may not add because of rounding. 

^1982-83 Pell awards totaled $2.4 billion. 
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table 20, the largest overaward amounts resulted from applicants • 
improperly identifying their status as dependents (they were 
chiefly dependent students who applied improperly as independent) 
and from incorrect reports of nontaxable income. The laijest 
sources of underawards were dependent applicants* incorrect 
reports of assets and home equity. The Department did not 
require the validation of any of these four application items 
in 1982-83. 



Table 20 

Frequency and Aggregate Dollar Effect 
of Errors m Nine Pell Application 
Items in 1982-83^ 



Recipients 

Item in error with error 

Dependency status^ 5.5% 

Other nontaxable 10.0 
income^ 

Household size 10.1 

Number in post- 5.9 
secondary schools 

Home equity 6.5 

Assets of dependent 18.0 
students 

Adjusted gross income 4.0 
of parents and inde- 
pendent students 

Income of dependent 1.5 
students 

Taxes paid 3.2 



Estimated error (million)^ 
Overawards Underawards Net 



$64 $ 0 $64 

46 0 46 

44 10 34 

32 9 24 

34 16 18 

39 22 17 

26 11 16 

13 1 12 

6 4 2 



^Data are from the Department of Education. The nine errors 
rank highest in the Department's findings of estimated net 
dollars awarded in error. Error is defined as a discrepancy of 
plus or minus $2 from the best award, except for dependency 
status. 

totals may not add because of rounding. 

^Of the 5.5 percent of recipients whose dependency status was 
incorrect, 5.1 percent were dependents applying as independent. 
There were payment consequences for only 3.6 percent; the other 
1.5 percent had no error even after correction, mostly because 
they were eligible for the maximum grant in either case. 

*rhe application calls for a worksheet calculation of many 
possible sources of income and the transfer of the total to 
the application form. The kinds of income inaccurately in- 
cluded most often in the composite were untaxed unemployment 
benefits, child support, other welfare, and noneducational 
veterans' benefits. 
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Validation may have been successful in correcting some 
misreporting of one of the two items validated in 1982-83, 
adjusted gross income • Net error in this item decreased from 
$38 million in 1980-81 to $16 million in 1982-83 (for savings 
of about $22 million). Table 20 shows that the item ranked 
seventh in net dollar error among other sources of inaccuracy 
in the applications. in 1980-81, it ranked third. 

The application item calling for the amount of federal 
income taxes paid has been overall a small component of student 
error, although the Department required that it be validat»'^. 
in 1982-83. In 1980-81, it ranked twelfth among other sources 
of inaccuracy in the applications, and net error in the item was 
less than $1 million. The 1982-83 data show that after valida- 
tion increased, the net error increased slightly in size and in 
overall importance; it ranked ninth among the sources of error, 
with net error totaling $2 million. 

In short, the Department has focused on student overawards. 
Two items, one ranked high and one ranked relatively low, were 
selected for validation in 1982-83, and more students were 
required to bring supporting documents for these items to their 
schools. The data show that net error in adjusted gross income 
did go down in 1982-83, the year of expanded verification. How- 
ever, the second item, net error in federal income taxes paid, 
was small in both years but increased somewhat in 1982-83. 
Other application items not included in the Department's verifi- 
cation requirements show sizable error rates, and several items 
showed increased frequency of error between 1980-81 and 1982-83. 

APPLICATION ITEMS INHERENTLY PRONE 
TO ERROR 

Not all the inaccuracies categorized as student error in 
table 20 result from clear-cut errors that could be avoided by 
students and parents in completing the original application. 
Some error discovered by comparing data on applications filed in 
1982 with "best values" developed by the Department researchers 
in 1983 included the failure to use data that were correct and 
available when an application was filled out. An example of 
this type of error is the submission of a bank balance that 
differed from the figure shown on official bank records. The 
applicant could have found out the correct figure and used it on 
the application. 

Another type of error may have resulted, however, from 
items that had to be completed with separate worksheets. The 
item called "other nontaxable income" — one of the four items 
with the highest rates of application error — illustrates this 
type of error and the difficulty of measuring it. The item re- 
quired the applicant's addition of up to ] 1 separate types of 
income such as child support received and untaxed portions of 
unemployment insurance. There was no room on the application 
form to show every component, but a worksheet was provided in 
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the application instruction package. The multistep process 
applicants followed in collecting the separate dollar figures, 
adding them, and transferring the total increased the chances of 
error. And since the worksheet was not submitted with the 
application, the original figures were not available for verif id- 
eation, if this were called for, or the Department's error 
research. An indirect method was used to estimate error in this 
item. 

A third type of error is very different from the one just 
described. Three of the four greatest application errors were 
in items for which applicants had to estimate data for a future 
period. A year or more after the applications were submitted, 
the Department's research found that these estiniates were often 
wrong. For example, to determine an applicant's status as a 
dependent, the 1982-83 application asked three questions about 
the extent of support received from parents in 1981 and repeated 
the same questions for 1982 # the calendar year not yet ended at 
the time of most applications. One of the three questions asked 
applicants to say whether they would be claimed as dependents on 
their parents' federal income tax returns for 1982, which did 
not have to be submitted until April 15, 1883. Applicants who 
made errors in the item on status as a dependent were found most 
often in error in the three prospective items. Similarly, items 
on the 1982-83 application form concerning household size and 
number of household members in postsecondary education asked for 
estimates of these figures for the period July 1, 1982, through 
June 30, 1983. Thus, an applicant submitting the Pell grant 
application early in 1982, for example, would have been making 
an estimate of household characteristics for a period as long as 
17 or 18 months in the future. 

In sum, the overall category of student error includes sev- 
eral different kinds of error. Some error stems from failure 
to use correct data, even though they were available at the time 
of application. Other error, more difficult to measure, can be 
traced to complex worksheets, which may be inherently prone to 
error. A third type of error, forecast error, may also result 
from items inherently prone to error and may be the least 
amenable to correction. Applicants projecting 17 or 18 months 
into the future may not be able to be 100-percent accurate. 

INSTITUTIONAL ERROR 

Institutional error in 1982-83 is shown in table 21 on page 
60. About two thirds of the total institutional absolute error 
consisted of overawards. Somewhat more pell grants with institu- 
tional error were underawards (18 percent) than overawards (16 
percent), but the overawards resulted in greater dollar error. 
Institutional errors decreased after 1980-81, when they were 
present in 42 percent of the awards, to about 34 percent. 

The major source of reduction in institutional error was a 
decrease in the absence from school files of a student pledge 
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Table 21 
Institutional Error in 1982-83 



Pell Grants^ 



Error 



Programwide estimates 
Error as 
Total error % of all $ 
(million) awarded^ 



Recipients 
with error 



Mean error/ 

recipient 
with error 



Underawards -?111 
Overawards 210 

Total $321 



5% 
13% 



17.8% 
15.7 

33.5% 



-$247 
528 

$379 



^Data are from Department of Education. Error is defined as a 
discrepancy of plus or minus $2 from the best award. Totals 
may not add because of rounding. 

^^1982-83 Pell awards totaled $2.4 billion. 



Table 22 



Frequency and Aggregate Dollar Effect of Four 
Institutional Pell Errors in 1982-83^ 



Recipients Estimated error (million)^ 



Error 


with error Overawards 


Underawards 


Net 


Missing financial-aid 


3.2% $95 


$ 0 


$95 


transcript 








Incorrect 








Determination of 


22.3 54 


94 


-39^ 


enrollment status 








Calculation or dis- 


12.3 40 


16 


24 


bursement of award 








Determination of cost 


10.8 14 


35 


-21^ 


of attendance 








^Data are from Department 


of Education . Error is 


defined as 


a 



totals may not add because of rounding 
^Net underaward. 



required by the Department, called a statement of educational 
purpose; the error declined by 89 percent from 1980-81 to 1982-83 
and is attributable to the Department's consolidation of forms. 
In 1980-81, an institutional error was counted if the stateraent 
of educational purpose that had been signed by a pell applicant 
was missing from an institution's files. In 1982-83, this sep- 
arai.e form was eliminated and the statement was incorporated into 
the student-aid report form (which a student had to sign in order 
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to receive an award), thus reducing the measured error. There 
is no reason to believe, however, that a student's adherence to 
the prescribed and sworn behavior — that is, to use the Pell grant 
only for educational purposes — would be more likely in either Oi. 
these procedures. Put another way, the reduction of error is not 
necessarily associated with any cost savings. In any event, in- 
stitutional error remained substantial. At $321 million, abso- 
lute institutional error constituted nearly half of the absolute 
total error in 1982-83 Pell grants. 

As shown in table 22, there were four different kinds of 
institutional error. Although it was infrequent, not having a 
recipient's financial aid transcript on file was a major source 
of overaward errors (since the entire amount of an award for 
which documents were missing was considered an error). The 
three other sources of institutional error were more frequent 
than all but one of the nine greatest sources of student error 
and they were associated with both overawards and underawards. 
The incorrect determination of r^nrollment status (full- or part- 
time) caused the largest total dollar amount of underawards. 

ERROR RATES WITH LESS STRINGENT 
DEFINITIONS 

The Department has no official definition of "accurate Pell 
grant award." The Department's two most recent research studies 
reporting error data used different definitions of accuracy. 
The 1980-81 study reported as errors awards beyond a $2 
tolerance level. That is, an award was considered accurate if 
it was within $2 of the correct figure calculated with the best 
data available from the research. The 1982-83 study reported as 
errors awards beyond several tolerance levels from $2 to $100. 
Both studies also presented supplementary calculations of error 
rates, treating as eligible the recipients whose files did not 
contain every required document. 

To test the sensitivity of the Pell grant error estimates 
to these two aspects of the definition of accuracy, we asked the 
Department to recalculate the figures shown in tables 18, 19, and 
21 with a less stringent definition than any used in the Depart- 
ment's reports. This alteriiative definition considered as eli- 
gible students from whose files documents were missing (although 
they may have had other kinds of error that were not counted 
before), and the definition considered awards accurate if they 
were within $100 rather than $2 of the amount calculated from 
best values. Expressed in terms of the average Pell grant of 
$959 for 1982-83, the $100 and $2 error tolerances represent an 
allowable degree of error of about 10 percent and less than 1 
percent, respectively. 

Using the less stringent definition of error reduced the 
proportion of recipients in error from 62.7 to 40.4 percent. 
(See appendix m for the complete recalculated tables.) The 
40.4 percent of cases with error under the revised definition 
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consisted of 27.1 percent of all recipients who received 
overawards and 13.3 percent with underawards. The estimate of 
absolute error for the entire program for 1982-83 under the most 
stringent definition was $649 million, but it remained ^t $530 
million even under the less stringent definition. 

As might be expected, the institutional error-rate picture 
changed when missing documents were not considered an error. The 
error rate shrank from 33.5 percent of cases to 17.2 percent, 
and underav/ards exceeded overawards in numbers and dollars. 

The less stringent definition also affected student error, 
reducing the proportion of cases in error from 39.4 to 27.3 
percent. The dollar amounts changed very little. As we noted 
above, students considered ineligible under the stringent defini- 
tion of error (counting missing documents as an error) did not 
have their application error counted; relaxing the document 
requirements permitted counting these new cases when they con- 
tained student error, which may account for the small decrease 
in the payment consequences of student error. 

We concluded from this analysis that even when a far less 
stringent definition of "accurate Pell award" is used, error 
rates and their dollar consequences remain substantial. Addi- 
tionally, underawards resulting from institutional error become 
an even more noticeable type of award error, affecting 10 percent 
of the student recipients. 

PROBLEMS WITH THE DEPARTMENT'S ERROR 
DATA AS GUIDES TO IMPROVEMENT 

Vie evaluated the Department's research and asked two 
independent statistical consultants to review the most recent 
report on 1982-83 errors. We found no reason to question the 
basic national data and aggregate analyses reported in the 
Department's most recent study of error, but our evaluation did 
reveal at lea t five limitations that affect the usefulness of 
the data and analyses for some kinds of dctdiltsd descriptions of 
the error -^roblem and, especially, for designing corrective 
actions. 

First, problems with the 1982-83 study's effective sample 
size mean that the Department does not have reliable estimates 
of error rates at different types of institutions. The Depart- 
ment used a method called "cluster sampling" to choose the insti- 
tutions from which to draw the 4,000 Pell grants for study. For 
technical reasons, a sample chosen this way can result in a small 
effective sample. A small effective sample may mean that there 
are very few examples of some kinds of students, institutions, or 
errors, and the chances of being inaccurate in generalizing from 
small numbers are much greater than when generalizations can be 
based on observations from the entire set of collected data, 
some kinds of corrective action cannot be confidently targeted 



without reliable estimates of error rates at different types of 
institutions. The Department's contractor performing the Pell 
grant error studies in 1980-81 and 1982-83 agreed with our obser- 
vations of the limitations associated with estimates for sub- 
domains but noted that the Department's budget for the research 
limited the precision that could be achieved. The contractor 
added that warnings about the limitations of the sampling method 
are included in a technical volume of its most recent report. 

Second, the Department's data on Pell grant error came from 
occasional research studies, not from regular program monitoring 
operations. Since the Department has not done an error study 
each year, it has no record of trends in error rates for the 
complete history of the program. Studies of error must compete 
with other priorities in the Department for discretionary funds 
in any year, and variations in the funds that are made available 
may result in variations in study design, which affects the 
comparability of various periods of error data. As we noted 
above, the 1978-79 error data are not fully comparable with the 
data in the most recent studies because of differences in 
methodology^ Inconsistent data limit both the design of 
corrective action and the evaluation of its effectiveness. 

Third, the studies did not examine special populations in 
the program. For example, there are no data for schools in the 
alternate disbursement system on error rates in institutional 
certification of eligibility or cost of attendance. Application 
errors by recipients at this group of schools were included in 
tne 1982-83 estimates of student error, because estimates of 
application error were not as expensive to obtain as estimates 
of institutional error, according to officials of the Depart- 
ment's research contractor. In 1982-83, Pell grants totaling 
$29.6 million were calculated and disbursed directly by the 
Departmen . to more than 32,000 students at these schools. Another 
example is the lack of information on error rates for the special- 
condition application. This is the Department's alternative 
application for students whose family situation has changed 
for the worse since the previous year. More than 204, OOC 
applicants filed the special-condition application in 198?* 83, 
and more than 116,000 students received grants that were based 
on data provided on this form. 

Fourth, the Department's research does not directly examine 
why applicants or institutions made the kinds of errors they 
did. This is a difficult task, in light of the diverse Kinds 
of error. The Department's studies emphasize identifying 
discrepancies and calculating their payment consequences rather 
than describing the behavior of applicants and institutions that 
caused the discrepancies. For example, the studies include no 
data on the extent of deliberate inaccuracy, or fraud. Yet only 
by being able to distinguish be-^-.ween fraud and the consequence of 
confusing application instructions, for example, can the 
Department consider how to create the conditions that allow 



accuracy and determine the chances for improvement with various 
corrective actions ,4 

Finally, the Department does not plan its studies to 
include a deliberate examination of the effects on error of 
promising institutional practices, such as locally developed 
quality control plans that might be considered for adoption 
throughout the program. The Department could better plan insti- 
tutional corrective action by knowing more about what is already 
working. 

In short, while the Department's two recent studies provide 
important evidence of national error rates, they have shortcom- 
ings. Some of these ^ould be offset in part by routine monitor- 
ing of the program's operation. But even the usefulness of 
these special research studies for developing ways of correcting 
error could be expanded greatly by using other designs and 
asking other questions. The questions would center on the 
reasons for error and the ways of creating the conditions for 
accuracy by both applicants and institutions. 



The Department' s studies, while limited in some respects, 
do identify continuing problems with award accuracy despite 
changes in validation methods. The error is sizable and 
persists. The Department has focused on reducing student error 
even though institutional error makes up almost half of the $649 
million of award error. 5 student error, which results chiefly 
in overawards, was present in 39 percent of awards in 1982-83 
and 38 percent in 1980-81. 

Of the two items selected for validation in 1982-83, the 
net error decreased for one, adjusted gross income. The other, 
federal income taxes paid, was a relatively small source of 
error both before and after the increase in validation 
activity. Error continues in the adjusted gross income item. 
Other application items not included in the Department ' s 
validation requirements show sizable error rates. Of these, 
three o:' the four application items with the largest error are 
forecast items, which may be inherently prone to error and not 
highly amenable to correction. Institutional error was 
reduced somewhat in 1982-83 by streamlining forms, but the 
reduction may not be associated with actual cost saving. 



^The Department has looked for correlations between error 
rates and some institutional characteriscics other than type, 
and it has investigated the characteristics of students 
associated with specific errors. These searches have been 
inconclusive. 

5The $649 million is absolute error, or the combination of 
overawards and underawards. 
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Throughout our analyses, we found that the Department* s 
research, while providing evidence of national error rates, has 
many shortcomings for guiding improvements. The research does 
not focus on the reasons for error and the possible changes that 
could be made to increase the accuracy of both students and 
institutions* 
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CHAPTER 4 

UNDERLYING PROBLEMS IN THE DEPARTMENT 



HINDER RESPONSE TO PELL GRANT ERRORS 

We examined three basic aspects of the Department's 
response to Pell grant error: the goals set for correcting 
error, the strategies selected for achieving the goals, and man- 
agement arrangements for supporting the strategies. The con- 
gressional request for this study asked about the goals, deci- 
sion process, evaluation methods, and other details of current 
validation, which focuses on some student application errors. 
However, since the Department's data show continuing problems of 
inaccuracy in application items not validated, as well as insti- 
tutional inaccuracies in processing Pell awards, we examined the 
Department's general approach to error beyond the specifics of 
validation in 1982-83. We found problems in goals, strategies, 
and management . A problem common to the three is lack of infor- 
mation. These problems help explain why error persists, despite 
the validation effort. 

THE DEPARTMENT LACKS EXPLICIT GOALS 
AND TARGETS FOR CORRECTING 
PELL GRANT ERRORS 

In addition to examining the goal of the Department's 
policy on validating applications, we looked for other goals for 
reducing error in the program, because the research has found 
several kinds of error from a wide range of causes that are 
potentially responsive to diverse corrective actions. For 
example, student application errors and institutional errors can 
both result in underawards and overawards, two different types 
of error that may have different remedies. Within the two over- 
all categories of student error and institutional error, numer- 
ous specific errors in individual application items or stages of 
institutional procedure can be chosen for attention. Each may 
have more than one cause, and the causes could be addressed in 
different ways. Thus, in choosing corrective actions, the 
Department faces many choices; goals are one way of aiding 
choices, along with data from past operations and any research 
or experiments bearing on the choices being considered. 

Once general goals have been set, targets must be chosen. 
For example, a target within the general goal of reducing all 
types of error might be to reduce the proportion of grants v/ith 
overawards and underawards of $100 or more from the current 40 
percent to 25 percent. An even more specific target might be to 
reduce the error rate in the institution; ' determining whether 
recipients are enrolled full-time or part-time from 22 percent 
to 15 percent. The targets could be for the short term, such 
as for a single year, or the long term, such as for a 5-year 
program. 
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We found that the Department has taken many actions con- 
nected in some way to the problem of Pell grant error. A par- 
tial list includes publishing regulations requiring on-campus 
validation, publishing a validation handbook to guide schools in 
doing the required work, changing from time to time the methods 
of selecting students for validation and the numbers to be 
selected, seeking additional funds from the Congress for expand- 
ing verification, contracting for studies of Pell error, train- 
ing institutional staff in following departmental rules and 
nianaging financial aid, and reviewing compliance with the pro- 
gram's rules at some schools each year. We found that these 
have not been guided by formal statements of goals, purpose, or 
philosophy and that no specific targets have been set. Depart- 
ment officials could not provide us any written statement of 
goals for validation or for other initiatives for increasing the 
accuracy of Pell grants. 

Therefore, we examined the general aims being pursued with 
corrective action. The major corrective action is validating 
student applications, which has the implicit goals of identify- 
ing inaccuracies by means of the campus-based verification of 
several data items and deterring further inaccuracies by means 
of giving publicity to verification. 1 The emphasis appears to 
be primarily on items on student applications that result in 
overawards. Thus, in considering a range of possible corrective 
actions, the Department has not emphasized activity directed 
toward correcting underawards and institutional error. 

In our discussions with Department and 0MB officials and 
with the contractors' staff who process applications and do 
research and analysis for the Department, we heard repeatedly 
the opinion that the action that has been taken could have bene- 
fited from more comprehensive discussion and clarification of 
goals, purposes, and targets, in light of disagreements within 
the Department and in the higher-education community. Officials 
told us that deciding on an action was often difficult and 
delayed because of basic unresolved differences over the accuracy 
of the error data, their meaning and implications, and the 
importance of increasing accuracy relative to other objectives of 
the Pell program. We heard about different objectives from 



iThe Department has tried the strategy of having the application 
processing contractor check for missing and discrepant data when 
applications first arrive. Under specific conditions, the De- 
partment returns an application and asks the student to examine 
particular discrepancies and correct them if necessary. Several 
items on the application such as data on Social Security bene- 
fits are also checked by computer-matching with government 
records. Both activities probably deter or correct some appli- 
cation errors. Because they have been used for many years and 
seem not to be under revision, we did not examine them with the 
same detail as wa did validation. 
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some officials, but most spoke only generally about the purpose 
of the Department's actions concerning Pell grant error* one 
official downplayed the results that might be obtained from 
correction on the grounds that the real objective is deterrence, 
or improving the accuracy of all applicants by increasing 
general awareness of the chances that their data will be 
verified. However, while we heard about efforts to detect and 
correct errors through validation, we did not hear about 
significant plans or activities for deterring Pell grant error. 

Although the Department has no formal targets, we found 
that it does estimate the potential reduction in net error from 
choosing the numbers of applications and application items to be 
validated. However, the data that are used to make this esti- 
mate are weak, and in 1982-83 they were misinterpreted (see 
appendix iv) . Thus, besides lacking explicit goals for its 
efforts, the Department does not have reliable forecasts of the 
likely effects of a given year's validation plan. 

The Department addresses institutional error through pro- 
gram reviews and training, although it has taken no specific 
action comparable in scale to expanding student validation. 
Officials in the Department's units responsible for reviews and 
training did not describe specific goals for improved institu- 
tional accuracy and, at the time of our review, had not been 
significantly involved in examining the results of the research 
on error in order to set goals for improvement. 

PROBLEMS OF STRATEGY INCLUDE MAJOR 
INFORMATION GAPS HAMPERING PLANNING 

We defined strategy as the plans made and activities per- 
formed to achieve goals. in addition to examining the Depart- 
ment's strategies for reducing Pell grant error, we reviewed how 
it uses data to evaluate and adjust its plans and activities. 

-S trategy on application errors 

The Department's strategy for reaching its major implicit 
goal is to reduce student application error by validation, which 
represents detection and correction after the fact. The Depart- 
ment has varied both the number of applicants it requires to be 
validated and the number of application items it requires insti- 
tutions to check for error. Although Department officials have 
discussed the benefit to be gained from a strategy of preventing 
erroneous data from entering the pell grant application system, 
in practice they have done little toward it. This is indica- 
ted, for example, by the absence of an aggressive prevention of 
inaccuracy by improving the application form, such as by reduc- 
ing or simplifying the data elements required on the form or by 
otherwise modifying the form and its instructions. We found 
only one limited test of an alternative form and modest use of 
expert consultants in forms design. Law and policy have set 
some restraints on change in the design of the form, but the 
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Department has rejected the consideration of some possible 
improvements, partly in order to avoid the disruption of 
established data-processing routines ,2 

Strategy on institutional errors 

The Department does not have a strategy for correcting 
institutional error other than regular program review and train- 
ing activities, and these activities have no formal goals or 
targets. We discussed with the officials in charge of program 
review and training whether their activities have reflected 
plans for reducing the institutional errors found in the Pell 
error research, officials in both areas could tell us little 
about the- ways in which their work is directly linked to the 
problems that are shown in the Dep?.rtment • s data. 

The Department's formal review of institutional compliance 
with the requirements of all federal stndent-aid programs is 
administered by the program review branch of its division of 
certification and program review. Performed in visits to 
individual schools, these reviews can be a strategy both for 
decreasing errors at a specific school and for developing fur- 
ther data on problems and their correction by analyzing the 
results of the aggregate reviews. We have commented before on 
the general limitations of the Department's program reviews for 
insuring that schools comply with program rules. 3 Only between 
48 and 57 staff members have been available in recent years to do 
the reviews, and, therefore, the number of schools that can be 
reviewed and the depth of each review are limited. The Depart- 
ment has reviewed 500 to 700 annually in recent years. Since 
5,000 to 7,000 schools participated in all the Department's 
financial aid programs in these years, about 10 percent ha 'e been 
reviewed each year. But not all reviews were on compliance: 
some were reviews of schools that had sought approval to offer 
federal aid for the first time, and others were reviews of 
schools that had closer! their doors. 

The typical visit is for 3 to 4 days by one person. 
According to a Department official, about 15 to 20 student-aid 
files are sampled for each program year being reviewed. The 



2 After we collected our data, the Department asked its forms- 
design consultants for a more comprehensive review of the 
federal student-aid application form. In an interim report 
to the Department on February 11, 1985, the consultants noted 
that only minor changes have been made in recent years, even^ 
though the form is complex and crov/ded, and suggested significant 
alternatives, including some consistent with our analysis in 
appendix VI. 

3 Many proprietary Schools Do Not Comply with Department of 
Education's Pell Grant Program Requirements , GAO/HRD-84-17 
(Washington, D.C.: August 20, 1984), pp. 31-33. 



review covers compliance with all program rules for financial 
aid, not just Pell grants, so that limited attention can be given 
to any of the particular types of institutional error found in 
the Fell error research. Time limits also seem to preclude a 
search in any depth for the causes of observed problems for the 
purpose of recommending corrective action. 

The program review official told us that there has been no 
written guidance on how to do a program review since 1978. 
Therefore, we could not determine whether the three Pell error 
studies have been reflected in directions to the reviewers for 
increasing their examination of institutional error. This offi- 
cial did not recall that any other, less formal guidance for 
program reviews had changed in response to the research findings. 

Department officials familiar with the scope and methods of 
the review process said they believe the Department already has 
difficulty determining compliance with one complex self-monitor- 
ing requirement that schools track the academic progress of aid 
recipients. If the Department required institutions to imple- 
ment quality control plans in response to Pell error, as has 
been proposed in circular letters to schools, it seems from what 
we learned from Department officials that present resources for 
program review would not allow for the effective review of the 
implementation or the results of such plans beyond the most 
basic questions on compliance. 

Department officials told us that there is no prescribed 
sampling procedure, such as random sampling; each reviewer 
decides how to select files for review. From a small number of 
files not selected at random, there can be no statistical basis 
for an accurate projection of error rates. The reviewers make 
quantitative projections of the incidence of problems and their 
dollar value from the records they examine, (Total institu- 
tional liability for errors in all the student-aid programs in 
fiscal year 1983, for example, was assessed at $6 million.) But 
m recognition of the data limitations, the Department usually 
offers a school the option of detailed study of larger samples of 
records, if the findings seem to be significant. 

Within the limitations imposed by the number of schools, 
the depth of review, and the accuracy of estimates, the Depart- 
ment's program review strategy is further limited in its ability 
to follow up its requirement that schools take corrective 
action. We presented our findings about this limitation in the 
report we referred to above. The schools are generally not 
revisited by Department reviewers to sea v/hether planned correc- 
tive actions have been taken, and the Department's reliance on 
independent biennial audits may not be a good monitoring tool 
because of their apparent weaknesses in coverage and in reporting 
compliance v/ith program requirements. 

Finally, the Department seems not to use its program 
reviews as a general source of information for evaluating and 
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understanding institutional behavior., A Department official 
told us that it has, for example, collected no summary informa- 
tion from review findings on either errors in performing valida- 
tions or difficulties with institutional complrance. Program 
review activity seems not to have been integrated into an over- 
all strategy: staff in the program review unit told us they had 
not seen the Pell grant error research, had not served in work- 
ing groups developing new validation policy, and had not been 
involved in developing draft regulations that included the 
possibility of substantial new quality control responsibilities 
for institutions and that should be reviewed. 

Besides reviews of compliance, the Department's strategy 
for reducing institutional error includes training and support. 
Officials at our case study schools told us that these could be 
useful, since Pell was the most difficult program to administer 
of all the programs in their offices. We found that the Depart- 
ment offers two kinds of help to institutions: voluntary work- 
shop training on all aspects of financial aid administration and 
the $5 administrative allowance for each recipient. Institu- 
tional accuracy could be improved with the availability of 
information on promising practices, but we found no specific 
materials for improving the accuracy of Pell awards. A new 
handbook on quality control has been written by a Department 
contractor, but it has not been adopted as Department policy and 
had not been distributed when we were doing our review."* 

We discussed training in financial aid administration with 
an official in the Department's training branch, and we reviewed 
course outlines. Much training time is spent keeping school 
officials informed of program regulations and procedures. Many 
topics are covered in the training in relatively short periods, 
so that it seems unlikely that Pell grant error and quality 
control issues could be discussed to any significant extent. 
Department officials told us that, except for a general inten- 
tion to reflect consistent exceptions or problems found in pro- 
gram reviews, there is no substantial link between the training 
activity and analyses of data on institutional problems that 
have been found in program reviews and the research. 

That is, the training curriculum has not based very much 
change on problems located in the field, nor has the Department 
encouraged specific institutions to take training when problems 
have been discovered. The Department does not use information 
on who attends the training to see if all types of schools are 
represented or to do other analyses. Several years of training 
evaluation data have been collected, but the data have not been 



^Department training officials told us in March 1985 that 
quality-control training workshops that were to use the new 
handbook had been canceled for the year because of a lack of 
funds. A new plan was developed to use the handbook at a few 
schools in a test of institutional quality control in 1985-86, 
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analyzed. Thus, training has not been part of a deliberate 
strategy for correcting or preventing institutional causes of 
inaccurate pell awards. 



problems with data for adjusting strategy 

We found many weaknesses in the information that is avail- 
able to the Department's officials for understanding the effects 
of its strategies and designing improvements. In general, feed- 
back on the performance of the pell grant system is voluminous, 
delayed, and ambiguous, and knowledge of the underlying behavior 
of applicants, recipients, and institutions is very limit jcI. 

To understand regular program operations, the Department 
examines reports received throughout the year from the 
application processor as applications are received and processed 
and from schools as payments are made to students. In addition, 
applicant and recipient files are merged after the end of a pro- 
gram year to allow analyses of the whole year. The production 
data are voluminous, and the staff responsible for producing the 
reports required of the previous processor expressed doubt that 
the results can be readily interpreted. For example, data that 
the processor provides on applicants at any particular point, in 
the year are especially ambiguous because the applicant "pool" 
is different at different times of the year. School payment 
reports provide nothing more than accounting data, so lifcle can 
be learned about the quality of operations at the schools. 

End-of-year reports are generally not available until long 
after a year has ended, and interpreting the data to evaluate 
specific strategies that might have been used during the year is 
difficult because of the mingled effects of changes in the pro- 
gram and changes in environmental and institutional conditions. 
Data from the special error research studies have the shortcom- 
ings described in chapter 3. 

Department officials and contract research staff tolc? us of 
numerous gaps in basic knowledge about applicants and institu- 
tions that make design and adjustment of strategy harder. For 
example, the Department knows little about how students and' 
their families actually fill out the application (when, where, 
with what available records and help, and with what use of the 
instructions) or about why they misreport data. Department 
officials have general knowledge about the basic institutional 
processes of administering federal aid, but they lack hard evi- 
dence from systematic research or monitoring to show which staff 
and other resources are available, the functions and priorities 
of financial aid offices that could be affected by federal 
policies, and the costs of the activities of these offices. 

The Department does use past data to simulate the budget 
effect of changes in the pell formula or other policy assump- 
tions. However, because of the limited knowledge of why people 
in the student-aid system act as they do, its simulations cannot 
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reflect the possibility that people will respond differently to 
changes in requirements or procedures. During our review. 
Department officials did not identify for us any systematic pro- 
gram of experiments with policy variations or pilot tests of ^ 
possible changes in aspects of procedure such as the application 
form. With such a program, the Department could try out promis- 
ing variations in real settings and add behavioral data to the 
simulations that would improve forecasts* For 1985-86, the 
Department plans two experiments, one with electronic data sub- 
mitted from schools and the other with institutional quality 
control . 

The Department has used the strategy of student validation 
for several years (the numbers validated in 1980-81 through 
1983-84 and estimated for 1984-85 are shown in table 23), yet we 
found many gaps in data for making decisions about it. We 
developed a list of decisions the Department must make regularly 



Table 23 

Selection Methods for Pell Application 





Number of 


applications selected^ 


1980-81 


1981-82 


1982-83 


1983-84 


1984-85 


51 


73 


1,395 


86 


152 


274 


246 


265 


287 


317 


e 


e 


e 


221 


191 


e 


e 


e 


454 


146 


e 


e 


e 


e 


206 


325 


319 


1,660 


1,048 


1,012 



Method 

Random choice^ 
Preestablished 

criteria^ 
Error-prone model^ 
Cross-year comparison^ 
QuestionabJe applicant 

status^ 

Total 

^Numbers are thousands. Figures for 1984-85 are targets estimated 

by the Department; figures for the other years are actual numbers 

selected. 
^As defined by the Department, 

set of selection rules that change from year to year and are 

based on how applicants correct originally reported data. 
^A set of criteria that are drawn from statistical analyses of 

applications containing error and that change from year to 

year. 

^Criterion not used. 

fused to select applications from students who also applied m a 
previous year; comparing prior and new applications leads to the 
selection of applications indicating suspicious data patterns. 

9several different kinds of criteria, including discrepancies in 
tax-filing details and applications indicating that the appli- 
cants are not U.S. citizens. 
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about validation, and we identified examples of related data and 
analyses that could form the basis for these decisions (see 
appendix IV). We found that the Department's data are inade- 
quate to support decisions on a number of dimensions, including 
selecting students for validation, forecasting the dollar costs 
or savings of validation, and evaluating its effects. For exam- 
ple, estimating the costs of administrative processes is often 
difficult, but the Department had little or no information, when 
It decided to expand validation for 1982-83, either on the cost 
and effort that institutions have spent in administering the 
Pell program or on how these might be affected by adding to the 
volume of validation. Without this type of information, the 
Department could not assess the reasonableness of the adminis- 
trative allowance of $2 per recipient that was proposed to the 
Congress when the expanded validation requirement was proposed, 
nor did the Department have data for estimating validation's 
burdens and other effects on students. 



Table 24 



Opinions of School Of^iciala on Propoaala 
for Changing PeAl Validation 



Proposa l 

Eliminate use of estimated data on the 

application 
rhorten application to 5 or 6 data 

itema 

Have proceaaor conduct validation by 
computer-matching application data 
with 2-year-old irs data 

Have applicanta send documents to 
proceasor for analysis rather than 
institutiona 

Change atudent-aid report tolerances 

Provide incentive for on-campus 
validation by allowing schools to 
keep and re-award a percentage of 
Pell funds saved through validation 

Validate 100 percent of Pell appli- 
cants on campus 

Base number to be validated at each 
school on school's error rate 

Validate a few itema for everyone, 
with intensive review of a smaller 
group 



Strongly 
■support 
cr support 

58 

45 

42 

39 



39 
35 



33 
23 
22 



Neither Oppose or 
aupport strongly 
nor oppose oppose 



19 
29 
25 

15 



49 
34 



10 
37 
24 



24 
26 
33 

46 



12 
32 



56 
40 
54 



On a final point, we note that clear and strong direction 
for adjusting strategy is not likely to come from postsecondary 
financial aid officials. In our national survey of institutions, 
we asked them to respond to various options for changing 
validation. The options came from Department proposals and 
knowledgeable people whom we interviewed. The majority of the 
officials supported only the option of eliminating the use of 
estimated data on the application (see table ^^). 
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MANAGEMENT PROBLEMS HINDER ACTION 
ON PELL GRANT ACCURACY 

We found that management problems also hinder the Depart- 
ment's improvement of Pell grant accuracy. Specifically, we 
found that the responsibilities for identifying inaccuracy, 
developing strategies to correct it, and implementing the strat- 
egies are fragmented. We found also that the Department does 
not plan or budget resources specifically for corrective action 
and that hasty decisionmaking about program administration is 
common • 

For example, table 25 shows divisions of the Department's 
office of student financial assistance and selected responsibil- 
ities related to Pell award accuracy at the time of our review. 
The diverse sources of Pell award error, the need for accurate 
and comprehensive data on error and the effect of corrective 



Table 25 

Selected Responsibilities for Pell Accuracy 
in Divisions of the Department of Education 
Office of Student Financial Assistance^ 

Division Responsibility 

po.xcy and program Development Analyze policy; develop cost and budget figures? write 

regulations; develop application forms and instructions 

Systems Design «nd Development Develop and oversee short- and long-term plans for all 

automated systems 

Training and Dissemination Administer training far financial-aid officials 

program operations Administer application processing contract; oversee 

school disbursements 

Certification and Program Administer audits, school certification, program reviews. 

Review and student validation 

Quality Assurance Gather information -'^ Pell grant error; develop 

correction plans and related materials 

^The table shows divisions of OSFA during the period of our review. In September 1984, 
the divisions of systems design and development, and quality assurance, were moved from 
OSFA to a new unit, the Debt Collection and Management Assistance Service, which is part 
of the immediate office of the Assistant Secretary for postsecondary Education. (See 
appendix VII . ) 



actions, and the range of potential preventions and remedies 
that could be needed suggest that all these divisions should be 
involved in coordinated work to analyze data, develop plans, and 
implement them. 

However, since the Department has no explicit goals and 
targets for reducing error in the program, it is difficult for 
the staff who are responsible for regularly maintaining the Pell 
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system under demanding conditions to decide how much attention 
to divert to the problem of error. Department staff told us 
that Its divisions have different perceptions of the definition 
and importance of error. 5 contractors who have worked with 
Department staff on Pell grant administration told us that a 
lack of coordination between divisions with fragmented responsi- 
bilities allows disagreement to persist unresolved for long 
periods. For example, we learned that several versions of an 
advanced statistical method called an "error-prone model" were 
developed to select applicants for validation, yet none was 
implemented until 1983-84. Several observers attributed this, 
at least in part, to unresolved internal dis.. jreement between 
supporters and detractors of error-prone modeling within the 
various divisions of the Department's office of student finan- 
cial assistance. 

^ _ The most serious example of apparent confusion over respon- 
sibilities was given to us by a Department official, who told us 
that the lack of coordination between the divisions responsible 
for generating and using data on error contributed to inaccurate 
or partial understanding of key research data, which in turn 
caused a major overestimate of the likely savings from targeting 
validatj.on on adjusted gross income in 1982-83. A report on the 
Department's research data on application errors in 1980-81 
awards showed a figure called "adjusted gross income error" that 
was estimated at a net total of $101 million. However, this 
figure combined two types of error in two different parts of the 
application: (1) using an incorrect adjusted gross income 
figure and (2) incorrectly claiming status as independent rather 
than dependent and, therefore, using an incorrect income figure. 

The report stated in a footnote to a table that the figure 
was a cc'i Jination, allowing the implication that validating 
adjusted gross income alone would not remove all the error 
included in the total figure reported. in discussing 
validation, however. Department officials routinely suggested 
that validating adjusted gross income for all students would 
yield about $100 million in savings. 6 According to a Department 

5we noted that institution and association officials knowlege- 
able about student aid beyond the Department also have different 
perceptions about how extensive the problem of error is in the 
Pell grant program. The Department's research data on 1980-81 
awards were controversial. The 1982-83 study had not been 
widely circulated at the time of our review, so we could not 
tell if it has generated similar controversy. 

6ln 1980-81, the $101 million in "adjusted gross income error" 
included $38 million of error made by those who correctly 
identified their dependency status. The remaining $63 million 
came from those making an error in claiming dependency status 
tand thus using the wrong adjusted gross income figure). As 
table 20 in chapter 3 shows, adjusted gross income error 
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official familiar with the decisionmaking that led to expanding 
valida'-ion in 1982-83, this misunderstanding about the multiple 
sources of error in the $101 million figure and limiting its 
correction to only one type of error resulted in part from the 
fragmentation of responsibility for gathering error data, inter- 
preting them, and developing policy responses to the findings • 

We found that the Department also does not plan or budget 
resources specifically for correcting Pell errors. Further, the 
Department does not analyze what is spent on validation. Thus, 
the Department cannot judge the cost-effectiveness of validation 
as one corrective action among many possibilities. 

When we examined the costs of validation — including the 
cost of the validations done by Department headquarters staff, 
the costs (included in the Department's overall contract with 
the processor) for handling the increase in corrections result- 
ing from validation, and other costs of administering the 
national validation effort (including the error research) — we 
found that the Departmenr's total validation cost was approxi- 
mately $5.5 million for fiscal years 1982 and 1983. Two compar- 
isons suggest that the scale of this expense is modest. First, 
although we did not gather data on all administrative costs in 
the Pell program, we found when we considered application-related 
costs such as the full cost of the processing contract, payments 
for the services of the multiple data entry processors, payments 
to institutions for administrative allowances, and the Depart- 
ment's costs for performing and overseeing validations that 
the latter made up about 8.3 percent of the 1981-83 total. If 
we were to use a more comprehensive list of Pell administrative 
costs incurred by the Department at headquarters and under con- 
tract (such as printing and distributing applications, operating 
public information services, preparing and mailing checks for 
students attending schools in the alternate disbursement system, 
and operating the data-processing system, that tracks advances of 
funds to institutions and payments to recipients), the Depart- 
ment's outlays for validation would be an even smaller fraction 
of the total administrative cost. Second, compared to the total 
funds awarded in student Pell grants in 1981-82 and 1982-83, the 
Department's outlays for validation are much less than 1 per- 
cent. (Further details are included in appendix V.) 

Plans and decisions concerning the details of Pell program 
administration are commonly made under very tight time con- 
straints, according to staff of the Department and the contrac- 

( narrowly defined) had decreased by 1982-83 from $38 million to 
an estimated $16 million, probably as a combined direct and 
indirect result of the expanded validation of that item. 
However, error in claiming status as dependent or independent 
remained almost the same, since validation procedures for 
independent students did not include a method of checking the 
accuracy of the claim of independence. 
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tors who are responsible for carrying out decisions. For exam- 
ple, only a few weeks elapsed between the fall 1981 receipt of 
the research data on errors in 1980-81 awards and a decision on 
how to respond to the complex evidence of various kinds of inac- 
curacy. The initial plan had been for the validation by the 
central application processor of a few data items for all appli- 
cants. The Department asked the Congress for authority to pay 
for this validation with more than $13 million from funds allo- 
cated to student awards. Officials of the 1981 contractor were 
given little time to consider the new idea. Many details of the 
Department s requirement were vague, yet cost estimates were 
requested quickly. The resulting estimates were very rough. 
Department officials told us that the time pressures in this 
period also contributed to the misinterpretation of error data 
and the inaccurate estimates of the likely effect of targeting 
only on aajusted gross income as discussed above. Staff in the 
Department and at the contractor told us that they recall numer- 
ous changes and considerable confusion when the Department dis- 
cussed its plans for expanded validation with the Congress in 
early 1982. (Table 26 summarizes the events from the Depart- 
ment s first request to final congressional action.) 

The results of this type of decision process included 
change m the directions given to the processing contractor (and 
the related costs of changing computer routines) and uncertainty 
for students and institutions. The Department did not notify 
the institutions of a settled validation policy for the 1982-83 
program year until June 1982, several months after eligible 
students began receiving notices from the processor to proceed 
with validation. Further, because final action came so late, 
the Department was not able .o completely carry out the direc- 
tion, ultimately set by the Congress in approving the supple- 
mental appropriation of $1 million for partial validation in 
1982-83, that the applications most prone to error be tarqeted 
for validation./ ^ 



^Very early in 1982, the Department told the processor to put 
aside a targeting model, based on statistical analyses of error, 
that had been planned for use in 1982-83 and instead to select 
100 percent of the eligible applicants as called for in the ini- 
tial Department proposal to the Congress. Selecting all appli- 
cants continued from March through mid- June 1982, when the De- 
partment told the processor to return to targeting because the 
Congress had not approved 100-percent validation for 1982-83. A 
congressional conference report directed the Department to tar- 
get validation to applicants "statistically most likely to be 
m error, but instead it targeted students more likely to be in 
error than randomly selected students, not necessarily the 
most likely. This method, "preestablished criteria" in table 
23, was based on judgmental analyses by Department staff, not 
statistical analyses of error-prone applications (see appendix 
VI). The same report directed that all applicants be validated 
m later years, which the Department has not done. 
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Table 26 



Chronology of 1981-82 Interaction Between 
the Department of Education and the Congress 
on Proposals to Expand Validation 



Date 



Event 



1981 
Dec* 8 



Dec. 11 



Dec. 15 
Dec. 17-18 



1902 
Jan. 22 



Jan. 27 
Mar. 23 

May 18 

June 10 

June 24 

June 28 
July 14-15 



July 18 



Secretary requests ainendinent to continuing resolution permitting $13.1 
million of Pell program funds to be spent on administrative costs of central 
validation by processor 

Senate staff tell secretary that it is too late to amend continuing 
resolution 

secretary requests authority to reprogram funds for central validation 

House and Senate cor .nittees reject immediate reprogramming; House schedules 
hearing 

Secretary modifies request, asVing for $10.2 million for 100-percent 
on-campus validation, since time does not permit adding validation to 
processor work load 

House holds hearing; Department revises its cost estimate for the published 
record 

House Committae on Appropriations reports urgent supplemental appropriations 
bill, disapproving reprised request for $1.9 million and 100-percent on-campus 
validation 

Senate Committee on Appropriations reports urgent supplemental appropria- 
tions bill, approving $1.9 million and 100-percent validation 

Conference approves $1.0 million and validation only of the most error- 
prone for 1982-83 and d.^rects 100-percent validation for 1983-84 and beyond 

President vetoes bill; the Congress passes revised bill. Department valida- 
tion funds unchanged 

President vetoes revised bill 

Conference issues i jport on further revisetf bill maintaining earlier report 
language on validation; the Congress passes bill. Department validation 
funds unchanged 

President signs Urgent Supplemental Appropriations Act of 1982 (Public Law 
97-216) includin^i $1.0 million for Pell validation 



The decisions on validation after 1982 have typically been 
made within a short period each year, making it harder to 
improve technical approaches within current policy or to consid- 
er major alternatives to validation. Decisions (such as numbers 
to be selected, criteria for selection, and items to validate) 
that occur in late fall are driven by the annual need to set the 
parauneters for the work of the application processing contrac- 
tor, which must be prepared to handle some five million applica- 
tions when they start arriving in February of the next year. 
The Department develops an overall system design, specifying all 
the processing steps, including the initial application checks, 
the rules for suspending applications when data are discrepant, 
and the calculation of eligibility with the latest formula. 
The design of validation for a coming year is a part of the 
overall design prepared in this short period, since the processor 
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must know the Department's criteria in order to select students 
for validation and print instructions on the student-aid report 
for the extent of the validation to be done in a year. The deci- 
sions about a coming year's validation are technically difficult, 
since they must be made when only partial data are available for 
evaluating the effects of validation approaches used in the year 
still in progress. Timely decisions are especially difficult in 
years in which the Department is negotiating with the Congress 
over the Pell grant formula. 

It is an issue whether the effort required to direct the 
operation of the currently complex Pell grant system allows 
staff enough time to consider alternative designs and strategies 
that could prevent or correct award inaccuracy. The conse- 
quence of devoting most of the effort, except for a brief period 
in each year, to regulc^r operations is modest progress toward 
resolving error problems (as shown by the data in chapter 3). 

THE DEPARTMENT DOES NOT ACTIVELY 
SCAN OTHER AGENCIES' EXPERIENCE 

We reviewed the role in the Department's decisionmaking of 
information on programs from other agencies that base award 
benefits on need. Other agencies that have a history of work on 
the accuracy of awards and the verification of applicants' 
data, and have tried specific approaches, might have had results 
the Department could learn from. We expected to find interagency 
discussion, in view of the current administration's initiatives to 
improve payment integrity across agencies. 

Few officials we spoke with described the Department as 
having made an active, continuing search for relevant experience 
elsewhere. For example, computer-matching income data on Pell 
applications with tax forms had not been explored with the 
Internal Revenue Service because conversations held several 
years ago led the Department to believe it to be unworkable. 
Similarly, Departmeni: staff responsible for training financial 
aid officers did not know of training strategies used by other 
agencies to improve quality control in organizations administer- 
ing federal programs. Experience with forms design developed in 
the public and private sectors had not been used very much in 
recent years in the Department. Nevertheless, the potential 
benefits of this approach were exemplified by the officials who 
were responsible for error research having commissioned a review 
of quality-control regulations in other agencies and having found 
apparently useful ideas. Moreover, we found, in discussions 
with Department staff, experienced observers of student-aid 
activities, and officials of other federal programs facing 
similar problems, many ideas that could be applied at different 
stages of the Pell grant process in an effort to improve its 
accuracy. (They are presented in appendix VI.) 

None of these can be described as proven remedies, but they 
promise to be useful and deserve further study as part of a more 
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experimental approach that could help the Department, The ideas 
we found raise many complex technical questions that are hard to 
settle with existing data. Pilot tests with ideas for corrective 
action allow systematic assessment of the costs and effects of 
changes such as redesigning application forms and computer-match- 
ing Department and IRS data. Experiments may be especially 
important if ideas are drawn from other programs; they are a 
plentiful source of ideas, but their circumstances are so differ- 
ent that their approaches should be given further study, develop- 
ment, adaptation, and trial in the educational context. 

Since Department officials noted that attention to the 
accuracy of Pell awards was spurred several years ago by pres- 
sure from^OMB's budget review staff, we spoke with 0MB officials 
to find out if their role had included sharing information from 
various agencies facing common problems. The officials we spoke 
with indicated that their role as part of the budget examination 
process had chiefly been to put pressure on the Department to 
take action on error — first to study it carefully and then to 
respond to the findings. It appears that 0MB offered little 
substantive guidance about promising practices drawn from other 
agencies. After we finished our review. Department officials 
told us that 0MB had provided the Department with structured 
guidance about the general elements of an acceptable approach, 
through a new management review process, but it is not clear 
whether the guidance was based on evidence of effective 
practices elsewhere or on general management principles. 

MANY POSSIBLE RESPONSES TO APPLICATION ERROR 
HIGHLIGHT PROBLEMS IN TAKING CORRECTIVE ACTION 

Table 27 is a list of alternative approaches the Depart- 
ment could take in response to application error, beyond modify- 



Table 27 



Alternative Approaches to Improve Pell 
Application Accuracy and Verif iability 



Approach 



Examples 



Change individual data 
elements 



Require names of household members and students in post- 
secondary schools 



Reqi:ire specifics of other income on form 



Redefine to improve verif iability (e.g., dependency 
items) 



Eliminate some kinds of income to simplify form 



Change method of gathering 
data from applicants 



Use different forms for independent and dependent and for 
simple and coiiiplex situations 



Enter data more directly into automated system 



Change need-analysis method 



Use fewer data items to determine need 



Eliminate prospective data items, using only base year 
to determine need 



o 

ERIC 



81 

m 



ing validation. If validation were included, our list of 
actions would be longer, and a variety of questions would have 
to be answered (as suggested in our analysis in appendix IV). 
Table 27 shows three types of change to the application process 
that respond to problems that were found from the Department's 
error research. Each action in the table could help correct 
some errors noted in the Department's research. For example, 
redefining specific data elements could make it easier to fill 
out the form or to gather information that would be easier to 
verify. Several forms could be designed, each for the use of a 
specific group of applicants, tailoring the instructions and 
data elements more precisely to an applicant's situation. 
Students whose family finances are not particularly complex 
could be given a simplified application form. The form could be 
simplified to require less information from students if the 
overall formula for calculating their need wert changed. 

However, making an informed '^"hoice among the possible 
responses is a difficult task that includes analyzing error 
data, developing the details of specific corrective actions, 
projecting the costs and benefits of alternatives, and comparing 
them against the various objectives of the Pell grant program. 
The choice is hard enough to make if corrective actions are con- 
sidered one at a time. If several actions are to be taken 
simultaneously, their interaction increases the complexity ot 
the analysis and the uncertainty of the outcome. 

We have observed that, during the period of our review, the 
Department had not planned corrective action far enough in 
advance of implementation deadlines to permit the study and 
deliberation that are required. An absence of goals and weak 
data complicate the effort, as does the lack of coordination 
among the Department's various offices responsible for the 
different aspects of the program. And even if all these matters 
were resolved, the focus would still be only on application 
error and would leave untouched the problems of helping institu- 
tions make their awards more accurate. 

UNDERLYING PROBLEMS OF GOALS, STRATEGY, 
AND MANAGEMENT MUST BE ADDRESSED BEFORE 
SEARCHING FOR TECHNICAL SOLUTIONS 
TO ERROR 

The choice of specific solutions to the problem of error 
can best be made, we believe, when there are explicit, clearly 
defined, and accepted goals. And, to the extent that overall 
accuracy is the objective, the goals must be comprehensive 
enough to include the correction of underawards and overawards. 
Because the many areas that require corrective action are com- 
plex, the overall goal may have to be broken down into interme- 
diate targets extending over a number of years. 

If specific goals have been established, various strategies 
for correcting error can be systematically considered. Strate- 
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gies can be considered that go beyond the remedial policy of 
validation and include an active search for ways to prevent 
inaccurate information from entering the system. Improved data 
would aid in both the design of strategies and the ^'valuation of 
their effects. A wide search for and deliberate study of the 
best practices and ideas from schools and other programs would 
be helpful. Experimentation with new ideas can offer useful 
pilot data before they are incorporated into the routine. 

Given the thousands of higher-education institutions with 
which the Department works to carry out the pell grant program, 
attention might be given to including postsecondary schools as 
it sets goals and develops strategies for accuracy. Their par- 
ticipation could be beneficial in providing the Department with 
additional data on institutional administration of federal stu- 
dent aid, helping it resolve differences of view on the extent 
and importance of Pell grant error, and building commitment to 
further corrective action, especially action that may be 
required of school officials. 

Finally, the Department could review its fragmented respon- 
sibility for action on the accuracy of Pell awards and adopt an 
approach that \/ould insure more timely and coordinated action to 
interpret data, develop a broad range of options, decide on the 
best ones, and implement and evaluate them. Department managers 
need to coordinate plans for corrective action that include ex- 
plicit analyses of the costs of the effort and that encourage a 
comparison of the costs and results of diverse strategies. 

We have identified some weaknesses in the evaluation of the 
grant program. The program would be better served by a monitor- 
ing and evaluation effort that begins with the goals of the pro- 
gram and proceeds to an examination of the implementation, 
costs, and effectiveness of the strategies for achieving the 
goals and that feeds information back for timely program 
improvement efforts . 

SUMMARY 

We found that the Department' s approach to the problem of 
inaccurate pell awards lacks well-thought-out goals and precise 
targets for guiding staff and institutions. The Department 
emphasizes only one of many possible strategies, lacks data for 
redesigning or improving validation, and is hampered by a 
management approach in which fragmentation and uncertainty 
persist in the offices with responsibility for the problem, as 
long as they are without a central policy direction. Decision- 
making under the continual stress of meeting annual deadlines for 
processing applications may make it hard for the Department to 
address the full range of error in pell awards and to design com- 
prehensive responses to it. 

Although many specific ways of preventing and correcting 
inaccuracy in pell grants can be suggested, we believe they will 
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have only a limited effect until the Department describes the 
nature and extent of the problem and follows this by setting 
explicit and comprehensive goals for correcting it. Both long- 
term and short-term, preventive and remedial, strategies could 
then be developed for reaching the goals. They will be imple- 
mented most effectively, however, by an office with well-defined 
and coordinated responsibilities for addressing Pell award error 
and for evaluating the results of adopting the strategies. 

The discussion of deciding whether to continue the present 
approaches or to adopt alternative ones raises the question of 
their feasibility. Increasing the quantity and quality of the 
data on the problem, analyzing a greater number of options from 
a wider set of sources, and spending more time deliberating 
their effects all have costs. As we have discussed, the contin- 
uing pattern has been to make decisions under serious limita- 
tions of data and time. Significant action will be required for 
change. 
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CHAPTER 5 



SUMMARY, CONCLUSIONS, MATTERS FOR CONSIDERATION, 



AND AGENCY COMMENTS AND OUR RESPONSE 



SUMMARY 



The pell grant program receives more federal funds than any 
of the five other student financial aid programs funded under 
title IV of the Higher Education Act. In 1982-83, the program 
granted more than 2.5 million students awards totaling $2.4 bil- 
lion. The administration of the program involves individual 
applicants and their families, a central applications processor, 
and institutions of postsecondary education. The discovery of 
substantial error in awards has led to increased controls in 
recent years, especially over student application data, culmi- 
nating in a major expansion of documentary verification of the 
applications of about 66 percent of the recipients of pell 
awards in 1982-83. In this report, requested by the Chairman of 
the Subcommittee on postsecondary Education of the House Commit- 
tee on Education and Labor, we have examined the effects of 
expanded validation on institutions and students, reviewed and 
evaluated the Department's data on the effects of validation on 
award error, and analyzed the goals, strategies, and management 
of the Department's efforts at corrective action. 

Effects of expanded validation 
on institutions and students 

We found that the Department's expanded validation require- 
ments have had some costs for schools and some modest effects on 
students. Validations done by schools increased 57 percent in 
1982-83, but some of the increase resulted from schools' volun- 
tary activity. 

Almost a third of the schools reported that their policy is 
to validate all Pell applicants. of the institutions not vali- 
dating all Pell applicants, 31 percent reported that they vali- 
dated more students than the Department required. Whether 
required or voluntary, increased validation activity required 
extra work, which schools accommodated by increasing the time 
staff devoted to validation by a third and by making other ad- ^ 
justments such as allowing overtime. However, few schools had to 
make these special adjustments to a great or very great extent. 
Validation continues to rank low among all student financial-aid 
administrative tasks in effort required, at least according to 
data from our case study schools. The institutions did have some 
problems with validation, but they were generally satisfied or 
neutral about most aspects and willing to see some form of vali- 
dation expanded to other federal student-aid programs. 

The cost of staff and other resources schools used in doing 
a validation appears to have averaged about $14. Validation 
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cost the most for small schools that cannot attain efficiencies 
of scale and for schools with short programs and, therefore, the 
frequent admission of new students without experience with 
financial aid. 

We gathered several different kinds of information to iden- 
tify the effects of validation on students. We compared the 
problems reported by Pell grant recipients who were and who were 
not selected for validation, as well as making comparisons within 
a small group of Pell applicants who never became recipients. 
We found that selecting applications for validation increased 
the proportion of students who changed their pell applications. 
We found also that the large majority of the students who were 
validated reported not having experienced adverse effects while 
some students did report problems. in addition, the institutions 
reported that about 5 percent of the validated applicants 
experienced problems that led them to defer enrollment, to change 
from full-time to part-time enrollment, and, in a very small 
percentage of instances, to leave postsecondary education. 

About a third of the students selected for validation re- 
ported delays in receiving awards; in contrast, about a tenth 
of the nonvalidated students reported such delays. Among stu- 
dents reporting award delays, about a sixth reported changing 
their academic plans (for example, taking fewer courses) and 
about two thirds reported making financial adjustments (for 
example, borrowing money or cutting back on their budgets). 
These effects may seem relatively small in the context of the 
millions of students who were validated, although even a small 
percentage of more than a million can mean that several thousand 
individuals experienced problems. in 1982-83, for example, an 
estimated 69,000 changed their academic plans to soiae extent as a 
result of validation. 

Effects of validation on Pell award error 

Validation was exp?.nded because of data showing error, and 
data on 1982-83 Pell grants show that despite increased valida- 
tion, substantial error continued. Totaling $649 million, 
including overawards and underawards, it was caused by inaccura- 
cies made by students and institutions, and it persisted even in 
student application items that the Department validated. In 
1982-83, the Department estimated that 63 percent of the Pell 
grants were inaccurate by $2 or more. (Forty percent were inac- 
curate when a less stringent error tolerance of $100 was used and 
several documentation requirements were disregarded.) 

The total of overawards and underawards resulting only from 
errors students made in their applications was estimated at $328 
million in 1982-83. The Department found that student applica- 
tion error occurred in 39 percent of the awards; in 1980-81, it 
had been present in 38 percent of the awards. Most (83 percent) 
of the student error resulted in overawards; however, an esti- 
mated $55 million should have been awarded and was not. Vali- 
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dation reduced error in one of the two application items that 
were verified — adjusted gross income. However, the two items 
verified in 1982-83, as well as other nonvalidated items, con- 
tinued to show discrepancies when "best values" were compared 
to the data provided by the applicants. Some of the discrepan- 
cies may result from items inherently prone to error, including 
items requiring forecasts. 

Institutional errors, which seem to result in a considerable 
number of underawards, have been given little attention by the 
Department. Institutional errors decreased after 1980-81, when 
they were present in 42 percent of the awards, to about 34 per- 
cent in 1982-83, but they remained substantial at a total of 
$321 million in overawards and underawards. Of this total, $111 
million, or 35 percent, were underawards; an estimated $210 
million was overawarded* Institutional error continued to total 
nearly half (49 percent) of the total dc xlar error in the Pell 
grant program. 

We found the Department's data on the incidence and dollar 
consequences of error useful at an aggregate national level. 
However, research limitations affect the usefulness of the data 
for some kinds of detailed descriptions of the error problem, 
especially for guiding corrective action. The research does not 
permit reliable estimates of error rates at different types of 
institutions, so that they cannot be targeted for either correc- 
tion or study of exemplary practices; it does not examine special 
populations; and it does not provide fully comparable data for 
examining trends. The research also does not focus in any detail 
on the reasons for error and therefore is not as useful as it 
could be in suggesting changes that might increase the accuracy of 
both students and institutions. 

Goals, strategy, and management 
of corrective actio n 

We found that the Department has no formal, explicit, gen- 
eral goals or specific targets for corrective action. The need 
for goals stems from the many choices the Department faces in 
deciding how to respond to the many opportunities for increasing 
the accuracy of Pell awards. The Department's main corrective 
action, student validation, has aimed at reducing error in 
selected f^pplication items that lead to overawcirds. Thore is 
little emphasis on either student error that leads to under- 
awards or institutional error. 

The Department's strategy has been remedial, to detect and 
correct some student application errors. It has taken little 
preventive action. It has done less to develop strategies for 
dealing with institutional error. The Department does formally 
reviev institutional compliance with the requirements of all 
federal student-aid programs, and it does offer voluntary work- 
shop training on all aapects of financial aid administration. 
This monitoring of compliance and training have not, however. 
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been used as direct strategies for addressing institutional 
error in the Pell grant program. 
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strategy formation and evaluation are additionally hampered 
by problematic data and the lack of data on current operations. 
There have been only a few pilot tests of alternative approaches, 
and there is no systematic program for identifying and experi- 
menting with practices that seem promising. When the Department 
decided on expanded validation for 1982-83, it had little or no 
information on the costs and other burdens for institutions or 
students. 

Management problems in the Department include the lack of 
coordination of diverse offices with responsibilities for aware 
accuracy. Decisionmaking for corrective action in 1982 and 
later was rapid, spurred by deadlines for application process- 
ing at the start of the annual cycle. For example. Department 
staff and officials of its contractors recall consideraole 
confusion in 1981 and early 1982, when responses were developed 
to the 1980-81 findings of error, in which there were uncertain 
interpretations of data, inaccurately high claims for the sav- 
ings that could be achieved, changes in plans presented to the 
Congress, changes of direction for the processor, and uncertain- 
ty about validation policy for 1982-83 for the applicants and 
schools. We found that Department decisionmaking cannot be 
based on direct study of the costs and results of corrective 
actions, since the Department has not explicitly planned or 
tracked its use of resources. 

We found that the Department has not commonly or systemat- 
ically gathered the experiences of other agencies with similar 
problems. We determined that 0MB, which might have helped inform 
the Department of other agencies* practices that promise to 
correct award error, had chiefly exerted pressure for action 
although it might have done more to share potentially useful 
examples with the Department. 

CONCLUSIONS 

In general, the Department has taken some useful steps 
toward Pell award accuracy; it has continued to study the extent 
of error, check applications through the application processor, 
and expand validation. But expanded validation has its costs, 
and error persists in the targeted items and items not targeted 
and in institutional processes not addressed. These observa- 
tions suggest that further attention is needed if pell award 
accuracy is to be improved. 

Many technical approaches could be examined in a search for 
ways to correct the kinds of error identified in t^^e research. 
(We suggest some in appendix vi; in api.c»ndix iv we discuss 
initiatives that could be taken for impr^oving validation alone, 
such as str ingthening and consolidating the methods for choosing 
applicants for validation.) However, our findings make clear 
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that choosing appro^^ches effectively and implemanting them in 
thousands of postsecondary institutions require setting goals 
and targets and searching for preventive strategies as well as 
remediation. These in turn require strengthened management in 
the Department, including the clarification of responsibilities* 

MATTERS FOR CONSIDERATION BY THE CONGRESS 

In view of the weaknesses we have reported, the Congress 
may want to consider the rates of Pell grant error to determine 
whether additional guidance to the Department would be helpful. 
An error- free environment is clearly not possible. However, an 
absolute error total of $649 million annually seems excessive. 

The Department has funded some studies to measure error in 
Pell awards, but we found many gaps in the evaluative informa- 
tion. The Department could but has not, for example, evaluated 
the reasons for student and institutional errors, evaluated the 
types of institutions that have high error rates to determine 
the assistance or remedies they might be given, or evaluated the 
effect of voluntary training programs. The Congress may wish to 
consider whether the evaluative information that is now avail- 
able is sufficient for achieving accountability and accuracy in 
the administration of the Pell grant program. 

MATTERS FOR CONSIDERATION BY THE DEPARTMENT 
OF EDUCATION 

Many initiatives could be taken to correct the various 
kinds of inaccuracy at several stages cf the Pell award proc- 
ess. Some initiatives would require changes in the legislation, 
but the Department could present proposals for legislative 
change and accompany them with data showing their likely 
effects. For example, the Department has recently proposed 
simplified need analysis. Other action can be taken within 
existing law and regulation. However, we do not encourage the 
Department to adopt new measu*.js until it has begun to address 
the underlying issues. 

Before taking specific actions, the Department needs goals 
for accuracy, diverse strategies clearly linked to the goals, 
better data, and internal management structures that will apply 
leadership to corrective action.! Until these steps have been 



After we gathered the data for this report, the Department 
developed a draft "Corrective Action Plan for payment Error 
Reduction for Title IV Student Aid Programs" in October 1984, 
as directed by 0MB. Several points raised in the plan are 
consistent with our analysis. it remains to be seen whether 
the Department's diagnosis of the problems will be expanded 
and internalized and whether remedial action will be vigorously 
and effectively pursued. The draft plan acknowledges the need 
for long-term and short-term actions. It acknowledges that the 
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taken, piecemeal approaches are likely to have problems of 
design, implementation, evaluation, and effectiveness. A com- 
prehensive e^^-^fort might include a more active search for 
relevant experience in other agencies and a greater use of pilot 
tests of promising practices without having to experiment on the 
regular system. In addition, a comprehensive effort to define 
goals and strategies might be made in cooperation with the post- 
secondary institutions that now carry the responsibility for 
day-to-day administration of the Pell grant program. This might 
be especially helpful if quality-control plans that would place 
even more responsibility at the institutional level are to be 
considered. 

MATTERS FOR CONSIDERATION BY 0MB 

We noted in an earlier report on issues in eligibility ver- 
ification in entitlement programs that 0MB could help combat the 
general problem of isolation and duplication of corrective 
action agency by agency. 2 qmb could promote the sharing of prom- 
ising internal practices in federal agencies, identify workable 
and effective approaches for institutions carrying out verifica- 
tion functions for federal programs, . and encourage the use of 
joint or substitutable eligibility in order to reduce the work 
required for need analyses and eligj.bility reviews wherever 
possible . 

We are encouraged by 0MB ' s requiring the Department to de- 
velop a plan for error reduction in the Pell program. We urge 
0MB to maintain its oversight of the program, in future manage- 
ment reviews, to insure that 0MB *s concerns and the issues we 
have raised, including the need for improvements in Pell grant 
evaluation information, are considered and acted upon. We have 
noted how the absence of information has hampered the Depart- 
ment's decisionmaking regarding Pell grants. 



current process is "systemically error prone" and that efforts 
at improvement have been "mechanical changes." For future 
action, the plan states that "corrective actions that fail to 
address the systemic delivery issues hold little promise of 
achieving decisive reductions in the existing error rate." In 
February 1985, 0MB directed the Department to implement most of 
the plan. In addition, in September 1984, the Department 
announced a reorganization that created a new debt-collection 
and management assistance service, reporting to the assistant 
secretary for postsecondary education, and that placed respon- 
sibility for quality control in federal student aid higher than 
before. 

^ Eligibility Verification and Privacy in Federal Benefit 
Programs; A Delicate Balance , GAO/HRD-85-22 (Washington, D.C.: 
March 1, 1985), pp. 47-48. 
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AGENCY COMMENTS AND OUR RESPONSE 

The Department of Education and the Office of Management 
and Budget read a draft of this report* Their comments and 
our responses are given in appendix VII • 
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COMMITTEE ON EDUCATION AND LABOR 



U.S. HOUSE OF REPRESENTATIVES 



S20 CAMNOK HOUSS OPnCS tUHCIKO 



WASHINGTON* D.C. 20516 



SUBCOMMITTEE ON POSTSECONDARY EDUCATION 



September 7, 1983 



Mr. Charles A. Bowsher 
Comptroller General of the 

United States 
General Accounting Office 
Room 7026 441 G Street, N.W. 
Washington, D.C. 20548 



Dear Mr. Bowsher: 

The Subcommittee on Postsecondary Education is preparing for reauthor- 
ization of the Higher Education Act, including the Pell Grant program 
of student financial assistance. 

This program has grown in the last decade, so that it now serves over 
two million students and distributes over two billion dollars each 
year. As a result of several Department of Education studies, con- 
cern has been increasing about errors in awarding funds in the Pell 
Grant program. In response to this concern, the Department developed 
a methodoligy known as "validation," which requires over one million 
of the applicants for Pell Grants to provide additional detailed 
documentary evidence to support their applications. We understand 
that the Department is collecting information on the effectiveness 
of this methodology as measured by the errors corrected this year 
through validation, and already intends to expand the procedure to 
other student aid progranis as well . 

Hov^ver, testimony to my Subcommittee and to the National Commission 
on Student Financial Assistance, has raised questions about the valid- 
ity of the Department's methodology, its costs, burdens, and other 
impacts and effects, though witnesses have lacked extensive data. 

I am, therefore, requesting that the GAO conduct an exploratory 
study to see whether these concerns merit more detailed examination. 
The Subcommittee is interested in obtaining information on the ques- 
tions presented in the attachment to this letter. 

Discussions between the Subcommittee staff and the staff from your 
Institute for Program Evaluation have indicated that the Institute 
would be able to provide us with a briefing and preliminary informa- 
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tion from your exploratory study by early next year. A written 
report could follow as soon as possible after that if we later 
decide that is useful. 

Thank you for your cooperation in responding to this request. If 
you have any q\iestions please have a member of your staff contact 
Bud Blakey or Maryln McAdam of the Subcommittee on 225-8881. 




Chairman 



PS/mmg 



attachment 
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<MecLions for Analycie ra:;ed on Exploratory Study 
Topic 1: Current policy 

(1) What is the goal of the current Education Department (ED) policy 
on validation of Pell Grant applications? 

(2) How did the Department decide on the current goals and methods, and with 
what consideration of burden? 

(3) What does it cost ED to carry out its current policy and methods of 
validation? 



Topic 2: Methodology 

{U} Does ED have reliable data on award errors on which to base policy, and have 
those data been interpreted using appropriate methods? 

(5) Are the methods for selecting students for validation statistically sound? 
Are they suited to the policy goals? 

(6) Does the Department evaluate its methodology and use the findings in 
regular improvement of its approach? 



Topic 3: Effects on institutions 

(7) What costs are incurred by the diverse types of institutions of 
higher education in doing the validations? 

(8) Are there other effects of the validation process on institutions? Are 
these more serious at peirticular kinds of institutions? 

Topic h: Effects on students 

(9) What are the effects of the process on students selected for validation? 

(10) Are there effects of the validation process that fall disproportionately 
upon particular groups of students? 



Topic 5: Alternative approaches and methodologies 

(11) What are various approaches to the problem of error in Pell Grant awards? 

(12) Are there experiences of other Federal agencies, or of the private sector, 
that offer useful suggestions on this problem? 

(13) Are there alternative methods for preventing or correcting award errors 
that could offer a better balance of positive and negative effects? 
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SAMPLING AND METHODOLOGY 

This appendix presents information about our samples of in- 
stitutions and students and our survey and statistical methods • 

THE SAMPLES 

Our study involved the three separate samples of institu- 
tions and four separate samples of students outlined in table 
28. Four of the seven samples were originally planned to allow 
generalization about particular populations or universes.^ 



Table 28 



Overview of Study Samples^ 



Sample 

primary institutions 



Supplemental institu- 
tions 

Case studies of school 
financial-aid admin- 
istrative costs 

1983-84 recipients at 
a community college 

1983-84 recipients at 
a private, 4-year, 
traditionally black 
college 

1983-84 recipients at 
a private university 
enrolling a national 
student population 

1982-83 applicants 



Planned population 

5,009 central or independent 
institutions participating in 
the regular disbursement system 

773 branch campuses in the reg- 
ular disbursement system 



If 532 primarily minority, low- 
income, urban students 

1,533 students 



469 students; the school vali- 
dated all aid applicants and 
more application items than 
required 

434,714 applicants found eligi- 
ble and highly in need but who 
never received an award 



Number in 

Planned Sample response respondent 
sample size Number Percent population 



400 

100^ 
12^ 

281 
269 

234 

300^ 



334 



49 



84 



49 



3,912 



146 
154 

124 

42 



52 
57 

53 

14 



788 
848 

248 



^Findings can be generalized to the population of respondents, which excludes all schools or 
students who did not respond to our survey. When we excluded nonrespondents , we took into 
account their probability of being selected for our survey. Hence, the ratio between the 
number of schools or students in a respondent population and the planned population may not 
be exactly the same as the »-atio between the planned sample and the sample responses. For 
instance, the respondent population for the primary institutional survey was 3,912, or 78 
percent of the planned population, but the sample response rate was 84 percent. 

bNot designed to permit generalization to any universe of institutions but intended to provide 
exploratory information on these populations. 

^Not applicable. 



^The 12 case studies of institutions were not designed to permit 
generalization to any universe of institutions* Rather, they 
were intended to provide detailed pictures of the administrative 
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Hence, they were designed to take into account important charac- 
teristics of the populations they were drawn from. For example, 
we made sure that our sample for the primary survey of central 
and independent institutions included public, private, and pro- 
prietary schools with programs of all lengths and that our stu- 
dent survey samples included both validated and nonvalidated 
students wherever appropriate* 

As is typical in survey research, the responses for the 
four samples fell short of 100 percent, but the structural plan 
for each sample was maintained. In other words, we were able to 
take into account the population characteristics we had planned 
for, in spite of having less-than-perfect response rates. Fur- 
ther, the response rates were adequate to permit generalization 
to most of the original population of interest. (We did not 
generalize to the nonrespondent portions of any population.) In 
addition, survey respondents appeared not to differ substantially 
from nonrespondents on important general characteristics. 

Primary; institutional survey; central 
and independent institutions 

Our sample of institutions for the primary institutional 
survey was stratified on two dimensions; type, or length of 
program, and control. There were four types of schools; 
schools with both undergraduate and graduate programs and 
4-year, 2- or 3-year, and less-than-2-year schools. There were 
three kinds of control; public, private, and proprietary. 

The sample was also weighted by the number of recipients. 
The number of institutions we sampled to represent each group of 
institutions, defined by a particular combination of type and 
control, was determined by the proportion of the total popula- 
tion of Pell grant recipients accounted for by that group in 
1981-82 (the most recent data available at the time of our sur- 
vey). For instance, 32 percent of the Pell grant recipients in 
1981-82 attended public institutions with graduate and under- 
graduate programs. Therefore, 32 percent of our planned sample 
of 400 institutions consisted of public institutions with both 
graduate and undergraduate programs. Although the sample size 
for each category of type and control was determined in this 
manner, the specific institutions sampled in each category were 
selected randomly. 

Our sample was designed to be representative of schools 
that participated in the regular disbursement system for Pell 



costs of financial aid at diverse institutions . Similarly, the 
samples for the supplemental institutional survey of branch 
campuses and the applicant survey were not designed to be repre- 
sentative in the sense of providing estimates for these popula- 
tions. Rather, they were intended to provide exploratory 
information. 
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Table 29 



Primary Institutional survey^ Sample; 
Responses by Category^ 

Type of control 



institution 


Public 




Private 


Proprietary 


Total 


5-year-plus 


112/128 


= 88% 


26/36 = 72% 


1/1 = 


100% 


139/165 = 


4-year 


21/24 


= 88% 


38/44 = 86% 


1/1 = 


100% 


60/69 = 


2- or 3-year 


95/108 


= 88% 


7/8 = 88% 


12/16 = 


75% 


114/132 = 


Less-than-2- 
year 


0/1 


= 0% 


1/1 = 100% 


20/32 = 


63% 


21/34 = 


Total 


228/261 


= 87% 


72/89 = 81% 


34/50 = 


68% 


334/400 = 



^For each combination of type of school, type of control, and 
marginal total, the numerator is the actual number of survey 
responses, the denominator is the planned sample size, and the 
percentage of responses represents the resulting fraction. 
The 66 nonresponses include 56 institutions that never 
responded and 10 that refused to respond. 



Table 30 



Primary Institutional Survey Sample; Differences 
Between Respondents and Nonrespondents 

Average Respondents Nonrespondents 



Number of 1981-82 recipients 
Number of applicants Department 
selected for validation 



(n = 334) 

956 
390 



(n = 66) 

801 
302 



Student-aid index 

Number of recipients with 
Greatest eligibility^ 
Least eligibility^ 



360 

456 
80 



288 

396 
95 



Cost of attendance 
Pell award 

^Student-aid index = 0. 
^Student-aid index « 1,600. 



$3,444 
$976 



$3,365 
$1,010 
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grants • (Schools that participated in the alternate disburse- 
ment system were excluded because they did not perform valida- 
tions.) From the 6,126 schools covered by agreements in the 
regular disbi..rsement system in 1982-83, we omitted 806 branch 
campuses because of problems with the Department's classifica- 
tions of these schools by type. Another 311 schools were 
dropped for various other reasons. Thus, our planned sample of 
400 institutions was originally designed to be representative of 
the remaining 5,009 independent or central campus schools in the 
regular disbursement system that were authorized and eligible to 
receive Pell funds in 1982-83. The sample was designed to allow 
us to make estimates for this population 95 percent of the time 
within a range of error of plus or minus 5 percent. 

The representativeness of the sample was affected by the 
rate of response to our survey. Table 29 shows that response 
rates were generally good for all combinations of type and con- 
trol and for the total sample. However, response rates fell 
short of 100 percent, so that when statistical adjustments were 
made to eliminate nonrespondents (in accordance with their 
probability of being selected for our survey), our actual 
respondent population was 3,912 central and independent institu- 
tions in the regular disbursement system. Our survey revealed 
that the 3,912 institutions enrolled 1,921,359 Pell recipients 
in 1982-83, or 76 percent of the 2,522,746 reported by the 
Department. 

Finally, as table 30 shows, the differences between 
respondents and nonrespondents were not great. For instance, 
the responding institutions tended on the average to have more 
Pell recipients than the nonresponding ones. This is not sur- 
prising, since the schools that tended to return surveys less 
frequently (proprietary schools with programs no longer than 2 
years) also tended to be the smaller institutions. 

Supplemental institutional survey; 
branch campuses 

When we designed this review, we did not intend to distin- 
guish between central and independent campuses and branch cam- 
puses. Our problems with the Department's classification of 
branches defeated this intention; nevertheless, we did not wish 
to omit branches entirely. Therefore, we drew a small supple- 
mental sample of branches and sent them the institutional 
survey. 

Prom the 806 branches, we dropped 33 for various reasons. 
Our sample of 100 branch campuses was thus designed to reflect 
the remaining 773 institutions in the regular disbursement sys- 
tem that received Pell funds from a central campus. The sample 
was stratified in terms of control and, like the primary insti- 
tutional sample, weighted by the number of recipients. For 
instance, 66 percent of the Pell grant recipients were enrolled 
at public institutions. Therefore, 66 percent of our planned 
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sample of 100 branches were public institutions* The response 
rates were not as good as for the primary institutional sample, 
but they were adequate for analysis. 

We analyzed the responses we received from the branch 
sample to see if they differed substantially from the responses 
we received from the central and independent institution sample. 

Public Private Proprietary Total 

41/66 = 62% 4/21 = 19% 4/12 = 33% 49/100 49% 

and we found that they did not. 

Case studies of the administrative costs 
of financial aid 

We made case studies of the administrative costs of 
financial aid at 12 institutions that were not representative of 
any population. Rather, they provided diverse examples of the 
resources required to administer financial aid. Because of the 
intensive research methods needed to review these resource 
requirements, we could not study a large and representative 
group of schools. 

We did not select these schools at random. Nine had been 
selected for case studies of the administrative costs of finan- 
cial aid in 1981-82 for the National Commission on Student 
Financial Assistance by Touche Ross. We contracted with Touche 
Ross to repeat the study of these 9 schools but for 1982-83 
costs. Joining the new data with the earlier data provided a 
comparison of costs at 9 schools before and after the Department 
changed its validation requirements. 

We included 3 additional schools in the study in order to 
increase the coverage of schools involved in the Pell program 



Table 31 

Characteristics of Case Study Schools 

Type of control 

Type of school Public Private Proprietary Total 



5-year-plus 3 2 0 5 

4-year 1^1^ i 3 

2-year or less 2^ £ 2^ 4 

Total 6 3 3 12 



^One school was added to the original group of 9 in 
order to compose the final sample of 12. 
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and thereby provide more information on 1982-83 costs • For 
instance, the original 9 had included only 1 4-year institution, 
a proprietary business school • We added 2 more, a private, 
traditionally black institution and a 4-year public college* A 
second proprietary school with a program of less than 2 years 
was also added, since many such schools take part in the Pell 
program* As table 31 shows, the 12 schools reflected the 
diversity of our primary institutional sample. 

Community college student survey 

The community college among the 3 student survey schools 
was also among the 12 schools constituting the case studies of 
the administrative costs cf financial aid. 2 The college serves 
an urban, low-income population made up mostly of minorities. 
We chose this school for our student survey primarily because of 
these characteristics. 

The community college had 1,532 Pell grant recipients in 
1983-84, and our planned sample was designed to be representa- 
tive of this group. The sample size for the community college 
sample, and for the 2 other student survey schools, was deter- 
mined primarily by trade-offs between our resource limitations 
and the response rates. On the one hand, our resource limita- 
tions allowed us to survey about 750 students in 3 schools. On 
the other hand, we were very concerned about response rates, 
which are typically no more than 30 to 40 percent for student 
samples. We did not want to sample fewer than about 230 students 
in any school. These concerns, and the particular structure of 
the community college sample (which was made possible by its ex- 
ceptionally comprehensive, computerized student-record system), 
led us to draw a sample of 281 community college students. 

The sample was stratified on two dimensions: financial 
need and validation status. There were two categories of finan- 
cial need: very high (a student-aid index of 0 to 200) and 
other (a student-aid index above 200). There were five catego- 
ries of validation status: not validated, validated and found 
correct, validated and found to have errors within tolerances, 
validated and found to have errors beyond tolerances but not 
great enough to warrant any change in award, and validated and 
found to have errors large enough to require changes in award. 
All recipients in the three last categories were included in our 
sample. A random sample of recipients was selected for the two 
other categories. 

As with the primary institutional survey, the final repre- 
sentativeness of the community college student sample was 



2a11 3 student survey schools participated in the case studies on 
cost, which provided comprehensive information about financial 
aid operations, including the validation procedures required 
of students. 
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H 
O 
to 



Student-aid Not 
indTex validated 

0-200 

200+ 

Total 



Community College student Survey Sample: 
Responses by Category^ 



Validated 



No error 



Error within 
tolerance 



Error outside 
tolerance but 
no award change 



28/50 


= 56% 


18/50 


= 36% 


12/26 


= 46% 


10/21 = 


48% 


28/50 


= 56% 


23/50 


= 56% 


1/4 


= 25% 


14/20 = 


70% 


56/100 


= 56% 


46/100 


= 46% 


13/30 


= 43% 


24/41 = 


59% 



Error leading 
to award cnanc^e 

1/4 = 25% 

6/6 = 100% 

7/10 = 70% 



Total 
69/151 = 46% 
77/130 = 59% 
146/281 = 52% 



^For each category and marginal total, the numerator is the actual number of survey responses the 
denominator is the planned sample size, and the percentage of responses represents the resulting 
fraction. ^ ij 
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affected by the response rate. Table 32 shows that the overall 
response rate was 52 percent^ and response rates were generally 
acceptable for all categories of financial need and validation 
status. 3 However, response rates fell short of 100 percent, so 
that statistical adjustments to eliminate nonrespondents (in 
accordance with their probability of being selected for our sur- 
vey) brought our respondent population at the community college 
to 788 1983-84 Pell grant recipients. 

Finally, as table 33 shows, the differences between 
respondents and nonrespondents were generally not great. For 



Table 33 



Community College Student Survey Sample; 
Differences Between Respondents 
and Nonrespondents 

Respondents Nonrespondents 
(n ^ 146) (n = 135) 



Average eligibility 416 313 

Number of recipients^ 

Dependent 79 68 

Independent 66 66 

^Information was missing for 1 respondent and 1 
nonrespondent. Thus, the number of dependent plus 
independent recipients in this table is 1 less 
than the actual total for each category. 



example, the average eligibility for nonrespondents was 100 
student-aid index points higher than the average eligibility for 
nonrespondents. However, given that the range of the eligibility 
scale was 0 to 1,600, both respondents and nonrespondents had, on 
the average, very high need. Further, of each group about half 
consisted of dependent students and half of independent students. 

Traditionally black institution 
student survey 

The traditionally black school was a private, 4-year school 
with a long history of enrolling blacks as the great majority 



3The response rates for all three student surveys were better 
than the response rate of 30 to 40 percent that is typical of 
student populations. 
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Table 34 



Traditionally Black College Student Survey 
Sample; Differences Between Respondents 
and Nonrespondents 



Respondents 



Nonrespondents 



(n = 154) 



(n = 115) 



Average eligibility 



479 



429 



Number of recipients 
Dependent 
Independent 



126 
28 



85 
30 



of its student body. We chose this school because of these 
characteristics and because of the special policy interests in 
the effects of federal initiatives on schools of this type and 
the educational opportunities of their students. 

This school had 1,533 Pell grant recipients in 1983-84, and 
our planned sample of 269 students was designed to be represent- 
ative of this group. As with the community college, sample size 
for the traditionally black school was determined primarily by 
our resource limitations and our concern about response rates. 
^The sample was stratified on validation status with its two 
levels, validated and nonvalidated students. ^ The final 
representativeness of the sample was affected by the response 
rate. The overall response rate was 57 percent (154/269), and 
the response rates of 55 percent (94/170) for validated students 
and 61 percent (60/99) for nonvalidated students were adequate 
for both subgroups, so the basic structure of the sample was 
maintained. However, since the response rates fell short of 100 
percent, statistical adjustments to eliminate nonrespondents (in 
accordance with their probability of being selected for our 
sample) brought our respondent population to 848 1983-84 Pell 
grant recipients. Table 34 shows some differences between 
respondents and nonrespondents, but they were generally not 
great. For instance, on the average, respondents and 
nonrespondents had very high need. 

Private university student sample 

The private university, serving a national population, has 
a long-standing policy of validating all its recipients of 



^The automated record-keeping systems at this school and the 
third school in our student survey were not detailed enough to 
permit the sample stratification we were able to use at the 
community college. 
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Table 35 

Private University Student Survey Sample; 
Differences Between Respondents 
and Nonrespondents 

Respondents Nonrespondents 

(n = 124) (n = 109) 

Average eligibility 491 469 

Number of recipients 

Dependent 95 72 

Independent 29 37 



financial aid, and it validates more items on Pell applications 
than the Department requires. We chose this school for our 
student survey primarily because of this policy. 

This school had 469 Pell grant recipients for 1983-84, and 
our planned sample of 234 students was designed to be represent- 
ative of this group. As with the two other schools, sample size 
was determined primarily by our resource limitations and our 
concern about response rates. The sample was not stratified, 
because the school validates all applicants for aid. Therefore, 
we drew a random sample of half of the 1983-84 recipients. The 
final representativeness of the sample was affected by the 
response rate, and we did not generalize to the nonrespondents. 
The response rate v;as 53 percent, and statistical adjustments to 
eliminate nonrespondents (in accordance with their probability 
of being selected for our sample) brought our respondent popula- 
tion to 248 1983-84 Pell grant recipients at this private univer- 
sity. Table 35 shows some differences between respondents and 
nonrespondents, but the differences were generally not great. 
Students in both groups were, on the average, highly in need of 
aid. similarly, both groups consisted primarily of dependent stu- 
dents although there were more dependent students among the re- 
spondents (77 percent) than among nonrespondents (66 percent). 

Applicant survey 

Our sample of 300 applicants was designed to provide 
exploratory information about the 434,714 applicants for 1982-83 
who were found to have the greatest eligibility for a Pell grant 
but never received one. The sample was stratified on validation 
status with its two levels of validated and nonvalidated stu- 
dents. The sample was divided equally betv/een these two groups: 
150 applicants had been selected for validation, and 150 had not. 

The sample was drawn according to our specifications by 
System Development Corporation, the Department's former Pell 
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grant application processor. We anticipated problems in finding 
respondents because we had planned to survey this group by tele- 
phone but the processor's records contained only a 2-year-old 
address list* Therefore, we asked for a random sample of 2,000 
high-need applicants, 1,000 selected for validation and 1,000 
not. Then we randomly drew a survey sample of 300 from this 
group, reserving the remaining 2,700 for substitutes for 
individuals in the sample of 300 who could not be located. 

We encountered even greater problems than we anticipated in 
finding telephone numbers and completing interviews with appli- 
cants. The sample of 300 was quickly exhausted, and when 
repeated attempts to locate 1,084 of the applicants from the 
first-stage sample of 2,000 yielded only 42 completed inter- 
views, we terminated the survey. This was not a sufficient 
number to permit any sort of generalization, even the most 
exploratory, but the 42 interviews did provide some information 
about eligible nonrecipients. 

SURVEY METHODS 

We developed three different questionnaires for our six 
surveys and tested them in the field. (Copies are available on 
request. ) One mail questionnaire was used for the primary and 
supplemental institutional surveys, and another was used for the 
student surveys at the community college, traditionally black 
school, and private university. The third was the telephone 
questionnaire. 

We made several attempts to reach the nonrespondents to our 
mail surveys and encourage them to return the completed 
questionnaires. For instance, we called institutions that 
failed to respond to the primary institutional survey three 
times. Similarly, we called students who failed to respond to 
the three student surveys at least three times, usually five or 
six times. 

All returned surveys were checked for errors and missing 
data before being coded and keypunched for computer analysis. 
Where necessary, we called respondents in order to correct 
errors and collect missing data. Coding was double-checked, and 
after the data were keypunched, we used the computer to check 
again for errors. 

STATISTICjUj METHODS 

All six survey samples were drawn randomly from their popu- 
lations. We used lists of random numbers to draw five of the 
samples but not the sample of students from the private univer- 
sity, for which we selected all 1983-84 Pell grant recipients 
whose Social Security numbers ended in an odd number • This 
procedure is as random as using a random-numbers list and has 
the advantage of saving time. 
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We weighted the data from our primary institutional sample 
and our three student samples in order to project sample data to 
each population and make estimates for and draw conclusions 
about the populations rather than just the samples. We computed 
confidence intervals for key population estimates, each interval 
providing an upper and a lower boundary for the population esti- 
mate, given the particular structure of the sample used to 
develop the estimate. (The intervals are available on request.) 
V7e took the size of the intervals into account in our analysis 
and interpretation of the data. 
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ERROR RATES USING LESS 
STRINGENT DEFINITIONS 

Table 36 

Absolute Error in 1982-83 Pell Grants 
Tr eating Cases with Missing Documents 
as Eligible and Using $100 Error Tolerance^ 



Sourco 

Institutions 
Students 

Total 



Programwide estimates 
Error as 
Total error % of all $ 
(million ) awarded'^ 



$202 
327 

$530 



8% 
13 

22% 



Racipients 
with error 

17.2% 
27.3 

40.4% 



Mean error/ 

recipient 
with error 

$463 
474 

$518 



^Data are from the Department of Education. Table includes all 
overawards and underawards added together^ without allowing 
them to offset each other within a single case or in the 
aggregate. Error is defined as a discrepancy of plus or 
minus $100 from the best award. Totals may not add because 
of rounding. 

^^1982-83 Pell awards totaled $2.4 billion. 



Table 37 

Student Error in 1982-83 Pell Grants 
Treating Cases with Missing Documents 
as Eligible and Using $100 Error Tolerance^ 



Error 

Underawards 
Overawards 

Total 



Programwide estimates 
Error as 
Total error % of all $ 
(million ) awarded^^ 



-$ 52 
275 

$327 



2% 
11 

13% 



Recipients 
with error 

5.3% 
22.0 

27*3% 



Mean error/ 

recipient 
with error 

-$389 
494 

$474 



^Data are from the Department of Education. Error is defined 

as a discrepancy of plus or minus $100 from t14e best 3:ward. 
^^1982-83 Pell awards totaled $2.4 billion. 
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Table 38 



Institutional Error in 1982-83 Pell Grants 
Treating Cases with Missing Documents 
as Eligible and Using $100 Error Toleranceg. 



Error 

Underawards 
Overawards 

Total 



Programwide estimates 
Error as 
Total error % of all $ 
(million) award ed^^ 



-$103 
99 

$202 



4% 
4 

8% 



Recipients 
with error 

10.0% 
7.3 

17.2% 



Mean error/ 

recipient 
with error 

-$411 
536 

$463 



^Data are from the Department of Education. Error is defined 
as a discrepancy of plus or minus $100 from the best award. 
Totals may not add because of rounding. 

^1982-83 pell awards totaled $2.4 billion. 
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DECISIONS ON VALIDATION AND DATA NEEDS 



To find out whether the Department uses evaluations of its 
validation methodology regularly to improve its approach, we 
looked at the available data and their role in decisions* We 
looked especially closely at the statistical soundness of the 
Department's methods of selecting students for validation. To 
begin, we dete-t^ined the major decisions that validation requires 
and, for each decision, developed a list of implied questions 
that could be answered by data and its analysis. For example, 
the Department has to decide which application items to validate. 
TO decide this, it would be useful to know what items are often 
wrong, by how much, and why, as well as the payment consequences 
of the errors and the likelihood that the verification of an item 
will reduce the error. 

Table 39 shows six decisions and the related questions that 
could usefully be answered by data and analyses. The decisions 
are what application items to validate, what evidence to require, 
vmat margin of error to tolerate, whom to choose to do the vali- 
dation work, how many applications to validate, and how to target 
applications if it has been decided not to validate them all. 
The decisions might at first be based on informal data and obser- 
vations but could be refined and adjusted as systematic research 
and evaluation data suggest more valid answers to the basic 
questions. 



Table 39 




Should be allowed? 



4. 



Who should do 
validation? 



What are the costs and quality of validation at alternative 
providers? Should effects other than costs be considered i; 



choosing providers? 



in 



5. 



How many applications 
should be validated? 



What are the costs and effectiveness predicted for validatii 
different numbers of applications and items? what is the 
relationship between number validated and the deterrence of 
error? 



6. If not all applications 
are to be validated, how 
should the targeted ones 
be chosen? 



What are the comparative costs (in time, analytic effort, and 
implementation) and the results of alternative selection 
methods? Do patterns in c-rrors permit efficient use of 
error* prone profiles as' a main selection method? 
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We discussed with officii'' s of the Department and its 
research and application-proce sing contractors how these 
validation decisions have been made and the data and analyses 
that were used in the decisions. Although on-campus validation 
has been required for some Pell grant recipients since 1978-79, 
we found extensive gaps in the information the Department has 
for improving the process. For example, to judge any particular 
set of decisions about validation parameters, the Department 
lacks strong predictive data on the costs of implementation or 
the likely effect on error. For another example, the Department 
uses four different criteria or sets of criteria to select 
applicants for validation and changes them frequently, but it has 
weak data to judge their relative effectiveness. 

In the rest of this appendix, we present our analysis of 
the data that were available to the Department in 3 984 for 
answering the questions on the right side of table 39. In 
general, the Department did not have systematic data that would 
fully answer most of the questions and hence based its decisions 
on the six issues on less formal information. The limited effect 
of validation may result in part from imprecise targeting, itself 
a result of incomplete data and analyses. 

THE ITEMS TO VALIDATE 

Choosing items for validation among the dozens on the Pell 
application is the initial decision in the eligibility verifica- 
tion process, and numerous criteria are relevant. The basic re- 
search data on application error that we describe in chapter 3 
show items that are often wrong and the nature of the discrep- 
ancies. The research permits a rank -ordering of application 
items according to the frequency of error and estimated 
overpayments and underpayments. However, validation is a 
questionable response to some kinds of discrepancies, and even 
where it is appropriate, it will not always produce dollar 
savings equal to the amount of error found in the research. 

For example, the items on dependency status are associated 
with the greatest consequences of error in terms of payments and 
are, therefore, an attractive target for validation, but there 
are numerous problems in verifying answers to the six separate 
questions on dependence. As presently worded, some are probably 
unverifiable (as discussed in the section below on sources of 
evidence) . 

Further, validating items that the research ranks high in 
terms of dollar error may raise unrealistic hopes, since the 
Department may require that schools use different information 
sources, at different times of the year, from those the research 
was based on. Different procedure?, will discover different 
degrees of error. The data the Department has do not go much 
beyond the basic error findings and are therefore of little help 
in analyzing the likely results of selecting given items for 
validation. (In chapter 4, we discuss the Department's signif- 
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icant overestimate of the potential savings from validating 
adjusted gross income in 1982-83 •) 

THE EVIDENCE TO REQUIRE 

Since validating an application item means comparing it to 
some sort of evidence, the Department must decide the sources of 
evidence to use and who is to obtain it, whether the processor, 
the schools, the students, or some combination of these. 
Typically, the evidence has been paper documents provided by 
students. Choosing sources of evidence could also entail 
evaluating alternative sources on criteria such as how easy the 
evidence is to obtain and use, its credibility, and the legal or 
privacy considerations that affect its use. 

The importance of evaluating alternative sources lessened 
in 1982-83, when the number of items to be validated was reduced 
to two, both verifiable from federal income tax returns. Finan- 
cial aid officials responding to questions in our national survey 
about problems obtaining and interpreting tax forms showed no 
widespread concern. However, validating other income and asset 
Items could be complicated and time-consuming for applications 
from students whose families have complex household fixiances, if 
financial aid officials take seriously the Department regulations 
that require schools to use all available information in examin- 
ing a student's entire application, including a full review of a 
tax return and its schedules, if these are submitted. Officials 
did report that documents beyond the tax form were often diffi- 
cult to obtain, although the students we surveyed at three schools 
did not report having this problem to the same extent. Before 
1982-83, a wider range of proof was required, but the Department 
had generally not gathered evidence to weigh alternative forms of 
documentation on the criteria we noted above. This issue will 
arise again if the Department decides to increase the number of 
validated items. 1 

The wording of some of the application items may allow no 
practical source of corroboration. For example, it appears that 
no evidence would be adequate to verify answers to some of the 
questions on dependence. Negative answers to the two Questions 
asking whether a student lived or will live with parents for 



^After we drafted our report, the Department published 
regulations requiring validation of six items in 1985-86. An 
applicant selected for validation must provide several kinds of 
evidence, but a signed statement is acceptable in several 
cases. The regulation and accompanying text do not explain the 
Department's choice of forms of evidence or the extent of error 
likely to be removed by requiring only signed statements for 
support of some items. Institutions can ask for more evidence 
beyond a signed statement if they have reason to doubt the 
student's information on several items. (See 50 Fed. Reg. 10710 
(March 15, 1985).) — ^ 
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more than 6 weeks and whether the student has received or will 
receive more than $750 for support from parents are probably 
unverifiable. Decreasing error by validating these questions 
may depend on changing the wording to ask, for example, whether 
the applicant maintains an independent residence for most of the 
year, the type of question that could be documented from rent 
receipts, utility bills, and the like. The Department's 
validation of this item through 1981-82 asked for no proof but 
simply signed statements from both applicants and their parents 
that the answers to the six questions on dependence were true. 
This requirement probably increased accuracy only modestly, 
especially since half the questions were about an entire year 
but the statements were collected in the summer or fall, 

THE MARGIN OF ERROR TO ALLOW 

Another decision is the discrepancy to tolerate between 
information on an application and the evidence used to verify 
it. Setting tolerances permits some error to go uncorrected, on 
the grounds that costs to applicants, schools, the Department, 
and the processor should be minimized where the payment conse- 
quence is small. Department officials told us that they had 
discussed among themselves the issue of tolerances for errors 
discovered in the course of validation and that they had argued 
with 0MB officials who believed that the established tolerances 
should be tightened. However, these discussions were without the 
benefit of adequate data and analysis and relied on impressions 
and general estimates of how much error could be permitted in 
application items being validated without affecting the amount of 
an award. 

Therefore, specific tolerances for individual items and for 
the overall application have been based on informal analysis, 
without adequately forecasting the effects of various margins of 
error. The study of 1982-83 error showed that these estimates 
were inaccurate and that current tolerances neither eliminate 
all inconsequential corrections nor require the correction of 
all consequential application errors. Had correction been 
required, the eligibility of 72 percent of the applicants with 
errors within tolerance would have changed, and two thirds of 
these changes v;ould have yielded net payment changes of §26 
million. 

WHO SHOULD DO VALIDATION 

Decisions on who should compare applications to supporting 
evidence should be based on data on the costs of the work and on 
any differences in the quality of validations by different organ- 
izations. In 1981 and 1982, as the greatly expanded validation 
plans were made (first for 100-percent validation by the pro- 
cessor and then for either 100-percent validation or less by 
schools), the Department had little data on Department, insti- 
tution, or processor costs. Since then, the Department has not 
made efforts to gather data on the cost, of the chosen approach 
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or its alternatives. The Department's research contractor for 
the study of 1982-83 error told us that the Department deleted 
survey items on campus costs, so that only general questions on 
school officials' opinions about effort and its burden remained 
in the survey. In our own survey, we found that such statements 
are useful but do not substitute for more reliable information 
gathered through detailed research methods. 

The Department lacks information on the quality of valida- 
tion also. The Department's error research has not examined 
validation practices in detail and has not been designed to allow 
precise estimates of either different error rates or the effects 
of promising practices at different types of schools. This type 
of information would be useful in deciding whether the schools or 
someone else should do validation and whether different types of 
schools should be given different instructions for the work. 
Since verification of eligibility and analysis of need are done 
in other federal programs, their experiences and data might sug- 
gest alternative approaches to the question of who should do 
verification, but we found that the Department has not explored 
these very much, believing that the educational setting of the 
Pell program makes it unique. 

THE NUMBER OF APPLICATIONS TO VALIDATE 

Since not all applications have error (see table 20 for the 
percentages of recipients with various kinds of application 
error), increasing the proportion of validated applicants may 
reach a point of diminishing returns. 2 to decide the proportion 
to validate, it would be useful to be able to predict the overall 
efficiency v including costs and results) of different versions of 
validation, considering different numbers to be validated along 
with variations in other parameters such as specific items to 
validate. However, the Department does not have an explicit 
model for deciding the number of applications to validate. 
Shortcomings in the available data on the costs of validation and 
on the effects of targeting particular items would make it diffi- 
cult to develop the integrated model that would be necessary. 

The practical decision is how many applicants the processor 
should select for validation, since the number of applicants who 
are finally validated cannot be controlled by the Department 
under the current design, over the years, the Department has 
used several different selection methods (see table 23). 
Department officials told us that, in the absence of a model, 
they do two rough analyses. Referring to past experience with 
the various methods of selection they have used, they predict 
the numbers that will be selected with the methods planned for 



Despite this, the Department did propose 100-percent 
validation for 1982-83, justifying it chiefly on the grounds 
that explanations to students and families would be simpler if 
everyone were targeted rather than selected individuals. 
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the coming year. They examine the predicted numbers informally 
to see if these will be acceptable to the higher-education 
community and to estimate, by rule of thumb, the dollar savings 
that can be expected. (The "rule" is based on observation of 
decreases in eligibility in past years after corrections have 
been submitted by applicants selected by each method. We evaluate 
this estimating tool in the next section.) If the dollar 
estimate "doesn't seem high enough" to senior department 
officials, then the number of applicants planned for validation 
may be increased, either by adding entire selection methods or 
by increasing the rate at which students are to be chosen 
randomly. This rate is also changed sometimes during a year, 
if actual selections are running below the projected figures. 

Random selection requires choosing a selection rate — for 
example, instructing the processor to select every fiftieth 
applicatio.i, which would be a 2~percent random sample. The 
Department uses the applications in this random group as a 
comparison group for studies of the other methods of selecting 
applications, and it believes that public awareness of the 
chance of random selection helps deter applicant error. However, 
the Department does not have strong data on which to base the 
specific choice of selection rate and total number to be chosen 
randomly. We found no statistical study designs cpecifying the 
random sample sizes needed for valid comparisons for evaluating 
the different methods and no data on the relationship between 
numbers selected for validation and deterrence. This weak 
rationale was the basis for the planned selection of 152,000 
applications for validation in 1984-85. 

APPLICATION SELECTION METHODS 

For 1982-83, the first time since validation began in 
1978-79, the Department proposed 100-percent validation, but the 
congress rejected the proposal, stating that the funds authorized 
for validation in 1982-83 should instead be "targeted toward 
validation of those applicants who statistically are most likely 
to be in error. "3 The Congress also directed that selection be 
replaced with lOO-percent validation beginning in 1983-84, but 
the Department has not done this and has continued to select only 
a fraction of the applications. We examined the statistical 
characteristics of the Department's selection methods and whether 
the Department has data to evaluate and improve these methods. 



3u.S. House of Representatives, Conference Report to Accompany 
H.R. 5922 , Report 97-605, 97th Cong. , 2nd sess. (Washington, 
D.C.: June 10, 1982), p. 5. H.R. 5922 v/as vetoed by the 
president, but the Department's provisions were included in a 
revised bill that was enacted, and the earlier report Ir.nguage 
on the provision was reaffirmed in the final conference report. 
(See Conference Report to Accompany H*R* 6855 , Report 97-632, 
97th Cong., 2nd sess. (Washington, D. C. : July 14, 1982), p. 18. 
See also table 26 for a chronology of the legislative events.) 
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If an application is internally consistent and 
arithmetically correct, there is no way to detect further errors. 
Thus, selection for validation is either a random choice or an 
exercise in prediction, an educated guess that an application 
contains error. The Department has developed a set of patterns 
or combinations of application data that it believes, given 
experience over the years, are suspicious and justify selecting 
for validation any application that matches a predetermined 
pattern, (The validation of applications is not linked to 
patterns that arouse suspicion but is, rather, for ths most part 
a uniform process for all selected applications in which items in 
a standard set are verified. Thus, the Department faces two 
separate sets of decisions in establishing selection methods and 
in setting the specific items to be verified on all applications 
chosen for validation.) We found statistical proolems with both 
of the Department's selection methods — random choice and the 
patterns, or "preexisting criteria." 

The Department's "random selection" method does not produce 
a true random sample for two reasons. First, students applying 
at different times in a year have unequal probabilities of being 
chosen if the Department has adjusted the selection rate (as 
discussed in the section above) because the processor's produc- 
tion statistics show that fewer applications are being selected 
than predicted (as can happen once or more during the course of 
a year). The resulting group is not truly random. Second, what 
is chosen if^. not applications but transactions. Each correction 
an applicant i^ubmits is considered a transaction, and the pro- 
cessor makes random choices from the set of all transactions. 
Applicants whose files contain multiple transactions therefore 
have a higher probability of being selected for validation, which 
also violates the rule in random sampling that all units have 
equal probability of being selected. 

In the other selection method, called "preexisting 
criteria," the processor checks applications against numerous 
separate data-patterns that have been developed by the processor 
and Department staff. The criteria are not based on statistical 
analyses of applications with error; they are drawn from informal 
analysis and hunches about suspicious combinations of application 
data. The list of criteria has been changed over the years, 
essentially by trial and error, by inspecting the corrections of 
eligibility and retaining criteria that are associated with 
reduced eligibility. 

Two selection methods used for the first time in 1983-84 
en-^ail "cross-year comparisons" and an "error-prone model." 
"Cross-year comparisons" are made only in considering the 
applications of students applying for a socond or subsequent 
Pell grant. The Department compares the prior application and 
the present one and selects for validation those that show 
suspicious patterns. As with the "preexisting criteria," the 
patterns observed between the 2 years may not, after all, depict 
errors. The method is based not on the statistical likelihood 
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of error but only on a presumption that the specified pattern 
(such as identical data for some item in both years) is likely to 
be a mistake* 

The "error-prone model" is the only selection method based 
on data showing the statistical likelihood that an application 
contains an error; it represents an approach we have encouraged 
in other agencies that use it. The Department's model is based 
on the findings from the error research, and it attempts to use 
all the data on an applicatio'^x (rather than just a few items, as 
with "preexisting criteria") in order to predict the probability 
of error. We did not review the adequacy of the particular model 
the Department uses. We have noted in chapter 4 that despite the 
model's stronger statistical basis, its use has been delayed 
because of disagreement within the Department over its value. 
The Department put the error-prone model aside in 1982-83 in 
favor of 100-percent validation. V7hsn the Congress rejected 
total validation and requested a targeted approach in midyear, 
there was not time to ready the processor's computers, so the 
targeting was done with the familiar "preexisting criteria." 
The Department began using an error-prone model in 1983-84 in 
addition to other methods, not to replace them. 

The Department lacks strong data to support its decisions 
about its selection methods. These decisions are made in the 
fall, when planning for the next year's application processing 
begins. The limitations of the decisions are the result of 
drawing conclusions from cross-sectional data and a weak measure 
of validation's comparative effect on nonequivalent groups. 

The "cross-sectional" data are weak because the year is 
still in progress when selection methods are evaluated in the 
fall, and information collected at that (or any) particular point 
in a year may be misleading: different students with different 
educational plans and financial aid needs apply at different 
times of the year. For example, staff at the Department's 
previous processor told us that applicants in the first part of 
the processing year are generally from 4-year colleges and 
universities that have a policy of selective admission or early 
deadlines for making decisions on financial aid for the coming 
year. Eligibility on the student-aid index tends to be lower 
among this group than among groups applying in other parts of the 
year, and their applications come primarily through the multiple- 
data-entry processors. Later in the year, applicants are more 
likely to apply directly to the Pell processor, seek admission to 
community colleges and proprietary schools, and rank higher on 
the eligibility scale. An evaluation of the effects of valida- 
tion will be misleading if it is drawn from data spanning only 
February of a year when processing starts through the early 
fall, when decisions on the next year's selection methods are 
made. 

The Department uses a weak measure to compare the results 
of the different validation selection methods* It would be 
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useful to know the dollar value of corrections: How much did an 
award change as a result of corrections stimulated by validation? 
And is there any difference in the degree of change among those 
chosen in different ways? But the Department does not know the 
payment consequence of any correction. It substitutes the more 
ambiguous ineasure of the change in eligibility observed after 
the correction of application data* The processor provides the 
Department with periodic tables showing, for each selection 
method, how many applicants have been selected and the average 
change in eligibility that has resulted from the corrections 
submitted by those in each group. For example, the Department 
might observe that there is a greater decrease in the eligibil- 
ity, after correction, of those selected up to that point in the 
program year by the "preexisting criteria" than for those 
selected by the "error-prone model." An observation like this 
might be taken as evidence that the former method is superior. 

Th^ limitations of this measure for evaluation are numerous. 
Most importantly, eligibility change is only a rough guide to the 
dollar consequences of a correction, since factors other than 
eligibility also enter into the calculation of an individual 
award, including the cost of the specific school attended and 
enrollment status. Further, the evaluation incorrectly assumes 
that the change in eligibility that is observed for each group 
can be attributed entirely to corrections stimulated by valida- 
tion. Corrections can occur for many reasons. An examination of 
eligibility change also does not reveal anything about whether it 
is validation of the specific items chosen in that year that is 
having the observed effect. The corrections that produce the 
change in eligibility may be of data items other than the ones 
chosen by the Department for that year's validation. 

There is, finally, one other reason why the Department may 
conclude improperly that one selection method is better than 
another, since the groups of applicants it compares are not 
equivalent. An accurate estimation of the result of using 
several different methods requires knowing how well the methods 
compare in selecting from a norm, or a typical group of 
applicants, but the Department dir jcts the processor to apply 
the selection methods in a specific order. For example, the 
Department may require the processor first to select all appli- 
cants who meet certain "preexisting criteria," then to apply the 
"random selection" methods, and finally to review the remaining 
applications using the "error-prone model." This means that 
the pool of applicants from which some are selected with a method 
given a lower priority will be different from the pool available 
for review with a method given a higher priority, which in turn 
means that the results of any comparison will be inaccurate to an 
unknown degree. 



Although the Department has useful data on the extent of 
error in specific application items, the decisions it has to make 
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on the design of validation as a response to errors are not sup- 
ported by strong data and analyses • For some items, analysis 
may show that validation cannot be effective, given the time of 
year and the types of evidence that are available when school 
officials carry out validation. The data that are available c^re 
incomplete, and in some cases misleading, for evaluating past 
validation decisions and for looking ahead to consider the advan- 
tages and disadvantages of options such as validating different 
items, requiring different evidence, applying other tolerances, 
or validating elsewhere than in the schools numbers of applica- 
tions selected by other methods. Our main conclusion is that the 
Department should consider other strategies beyond validation for 
their potential ability to curb pell error. The current valida- 
tion strategy could clearly be strengthened by improving the 
data and analysis. 
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THE COSTS OF VALIDATION 



The Department spent approximately $5.5 million on valida- 
tion and related activities for the period 1981-83. Adding to 
this the average institutional cost of $14 per validation for 
the 1*98 million students selected for validation in 1981-82 and 
1982-83 would make direct and indirect costs of validation total 
less than 1 percent of the amount awarded in Pell grants in 
these 2 program years, or an even smaller percentage of the 
total cost of the overall Pell program. In 1982-83, at an aver- 
age cost of $14 each, the 1.66 million validations the Depart- 
ment required cost schools an estimated $23 million. 

This estimate of $5.5 million takes into account the 
Department's costs at headquarters for performing validations 
for students attending schools in the alternate disbursement 
system, the administrative costs (planning, research, training, 
program reviews, and the like) associated with overseeing the 
validation that schools in the regular disbursement system do, 
and the Department's costs for validation tasks done as part of 
the application-processing contractor's work (see table 40). 

Our data on the Department's headquarters expenses are for 
October 1, 1981, through September 30, 1983. This period 
includes most of the Pell grant program years 1981-82 and 1982- 
83, or the year before and the year of expanded validation. The 
number of validations done at the Department increased from 
2,764 in 1981-82 to 12,885 in 1982-83. The Department's costs 



Table 40 



The Costs of Pell Validation at the Department 
of Education and the Processor in 1981-83 
(in Thousands)" " 



Cost 



1981 



1982 



1983 



Total 



Headquarters^ 

Performing validations 
Administering national 
validation policy 
Total 



$681 $1,014 $1,695 



$205 $ 295 
476 719 



Processor^ 

All validation-related 
activity, indirect 
costs, and fees 



$531 $2,237 $1,046 $3,814 



TOTAL 



$5,509 



^Data are for fiscal years. 
^Data are for calendar years. 
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also increased, largely because of the expenses of the research 
study of errors rather than operating costs in performing vali- 
dations. To obtain the Department's costs, we asked the Depart- 
ment to provide figures for various specific expenses of perfor- 
ming validations and overseeing schools' validation. We accepted 
the Department's estimates of staff time and statements of 
salaries and other expenses. We computed personnel costs from 
data the Department provided on salary and time estimates. 

Our data on the costs the Department incurred at the appli- 
cation-processing contractor cover calendar years 1981-83, a 
longer period than for the Department headquarters data. Calcu- 
lating the processor's costs required both direct and indirect 
methods. Expenses such as the costs of programming computers to 
carry out the Department's selection criteria and producing 
management information reports on validation selections are 
clearly attributable to validation. The main cost of validation 
is the processing of corrections that school officials require 
after reviewing the validation documents. Since we had no direct 
way of determining the volume of corrections that resulted from 
validation alone, we developed an indirect method. 

We requested that the processor sample several hundred cor- 
rections in each of 3 program years and calculate for each sample 
the fraction of corrections that came from applicants who had 
been selected for validation and that were dated after the 
notification that they had been selected. We assumed that all 
corrections submitted after the notification date resulted from 
validation (or from the warning that it would be required before 
granting awards). In 1981, only 4.9 percent of the corrections 
were of this type, according to the processor's sample study; 
the proportion rose to 26.1 percent in 1982 and fell to 14.1 
percent in 1983. We directed the processor to use a sample of 
corrections in each year large enough that the sample results 
could be projected to the total volume of corrections (within the 
limits of statistical confidence) for an overall estimate of 
the validation-related work load in each year. We gathered data 
at the contractor's offices to estimate the overall costs of 
processing all corrections in each year, and then we used the 
fractions described above to calculate the portion of those costs 
attributable to validation. 

To obtain a figure for the cost of the program that we could 
compare to validation costs, we added the Department's total 
outlay for student awards, payments to institutions for the 
administrative allowance, the total price of the application- 
processing contract, and payments for the services of multiple- 
data-entry processors. V7e did not estimate the Department's 
overall administrative costs for the Pell program. Doing so 
would have made validation's fraction of the program costs even 
smaller. 
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ALTERNATIVE PRACTICES FOR MAKING 
ACCURATE AWARDS 

In this appendix, we outline current practices at six stages 
in awarding Pell grants and discuss alternatives that the Depart- 
ment could explore. The six stages, shown in table 41, cover the 
Department's basic activities of defining need, obtaining data on 
individuals, verifying that the data are accurate, and working 
with the higher-education institutions that carry out administra- 
tive tasks for federal student aid. Alternative practices are 
possible throughout this sequence.! We focused on application 



Table 41 



Step 

Need analysis 



Data required for 
need analysis 



Application methods 



Data verification 



Resources and support 
for institutional 
effort 



Oversight of 
institutions 



Pell Grant System Practices and Alternatives 
That Could Improve Award Accuracy 

Practice 

Need determined from data on 

past year and estimates of 

year for which award is to 

be nade; 
Educational need stands alone, 

not linked to other benefits 

Maximum data gathered for char- 
acterizing family situation, 
resulting in complex data not 
necessarily high in quality 

Single paper forn for an 
applicant; 

Uses unique terns and defini- 
tions ; 

Asks for aggregate data; 
Mailed to processor 



Alternative 
Use base year only; 



Substitute eligibility for 
other need-tested benefits 

Simplify need analysis to use 
fewer and more reliable data 
elements 



Selective (students, items); 
Done at schools; 
Comparisons of data on paper; 
Before first installment of 
award 



Different forms for different 

situations; 
Use standard definitions (e.g., 

IRS terms) ; 
Ask for specifics; 
Direct entry to automated 
system 

More comprehensive; 
Done centrally; 

Comparisons of data by computer; 
After first installment of award 

to permit correction by second 

installment 



Administrative allowance with- 
out clear basis for amount; 

Training gives modest attention Explicit training on award 
to issues of quality quality control 



Explicit cost sharing based on 
formal study; 



Limited program review; 
Occasional national research 

studies; 
No goals or targets nor link 

to incentives 



Local monitoring; 
Regular data; 

Establish error-rate targets 
linked to incentives 



^We discussed 13 issues about determining eligibility for federal 
benefits and alternative approaches to them in Eligibility Veri- 
fication and Privacy in Federal Benefit Programs: A Delicate 
Balance , GAO/HRD-85-22 (Washington, D.C.: March 1, 1985) . 
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and verification activities since applicant error was the main 
subject in most of our review. We did not develop a complete 
analysis of current and alternative practices for other stages 
the schools perform, such as reviewing nonfinancial eligibility 
criteria, calculating and disbursing awards, monitoring academic 
progress, and making refunds. 

ANALYZING NEED 

The basic policy choice in designing a method for analyzing 
a student's need for financial aid is the complexity of the data 
that are to be used. Considering data on more factors and being 
able to measure them closer to the period of enrollment in post- 
secondary school may mean a more accurate estimate of need. But 
the same qualities that make information more sensitive to family 
differences and how they are expected to change may make the 
analysis more vulnerable to measurement error. For example, 
computing need from data on circumstances expected for the year 
in which a grant will be used, rather than from data on the 
previous year, allows for anticipated financial changes from the 
expected birth of a new family member or the first- time enroll- 
ment of several family members in school at once. But the effort 
to increase the sensitivity of the analysis by using forecasts 
creates the possibility of greater error in making awards, since 
a forecast may prove wrong. 

The Department's current practice is to combine data from a 
past "base year" with estimates about conditions that will affect 
an applicant's family expenses and resources during the period 
in the coming year when the award will be used. For example, an 
applicant for aid for 1984-85 would have been asked during the 
early months of 1984 about financial details of the fully 
completed year 1983 as well as conditions expected in 1984 and, 
for some topics, through June 1985. 

The forecast items the Department asked for included three 
questions on dependence: whether the Pell applicant would live 
with parents for more than 6 weeks in the full calendar year 
1984, would receive more than $750 in support from them, and 
would be claimed as a dependent on a parent's 1984 federal income 
tax form to be filed in 1985. Household size was also to be 
estimated for July 1984 through June 1985, and so were the number 
of family members who would be in postsecondary schools during 
this period and income and other financial details for the 
parents and students for 1984 and part of 1985. The Department 
required these data in an attempt to determine need from 
comprehensive figures . 

The Department's research data on error rates, however, sug- 
gest that, if the objective of more accurate awards is to be 
emphasized, one alternative is to reconsider such data elements 
and the broader policy of making comprehensive need analyses from 
multiyear data. Of the four greatest sources of inaccurate data 
(with payment consequences) in 1982-83 applications, three were 
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the partly or completely prospective items on dependency status, 
household size, and number enrolled in postsecondary education 
(see table 20). Problems of proof for these items make it 
uncertain whether increasing validation will correct them. 2 An 
aid in reconsidering the current approach would be to analyze 
various existing data bases on Pell applicants and recipients 
to see what changes in awards would be likely if need analysis 
included only data from the preceding year. 

A second alternative is to simplify the determination of 
eligibility for Pell awards by allowing eligibility for some 
other benefit program to substitute for a separate determination. 
There is precedent for this in the recent decision of the Food 
and Nutrition Service of the Department of Agriculture to omit 
verifying the eligibility of applicants for free and reduced- 
price school meals who can provide a number showing that they are 
eligible for the Food Stamp Program. Technical analyses are 
needed that would consider the problem of using the eligibility 
determinations that are made monthly or quarterly in different 
ways by different programs, since Pell is an annual program. 
However, exploring the alternative of using common eligibility 
seems reasonable in view of the extensive efforts other benefit 
programs make to gather and verify data in order to assist low- 
income families, some of whom include individuals who are likely 
to apply for educational assistance. 

REQUIRING DATA 

Views are diverse on how extensively information should be 
gathered, even if need were to be analysed from data for only 1 
year. Officials of the application processor and others in 
financial aid told us that they believe campus officials 
generally prefer to see as much data as possible about appli- 
cants, even though those data vary greatly in quality. The ob- 
jective of gathering data from applicants is, of course, to make 
awards that meet the students' needs as much as possible by 
taking into consideration as much detail as possible on family 
circumstances — income, assets, expenses, and so. Many data 
elements in the Pel\ grant formula and, therefore, on the 
application are set by law. 

However, in light of the continuing problem of inaccurate 
awards, an alternative is to consider simplifying the requirement 
for the data that are used in the analysis of students' need. 
This would permit simpler application forms and instructions, 



ERLC 



^Dependence might be verified most easily by examining the 
federal income tax returns of the parents of independent 
applicants to see if they are claimed as dependents. The 
Department believes it now has legal authority to do this in 
certain cases only. This issue could be clarified by the 
Congress in the legislative reauthorization. 
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which in turn might elicit more accurate information from the 
applicants and their families. Eliminating prospective data is 
only one kind of simplification that could be considered. 

Nontaxable income could also be examined from this per- 
spective, since it is a significant source of inaccuracy (see 
h table 20). The instructions are complicated for this item: for 

the 1984-85 application, up to 18 blanks had to be filled in on a 
separate worksheet, and a total had to transferred to the main 
part of the form. We suggest this data element be reviewed 
because of its complexity and the history of error associated 
with it. A review of a variety of income and asset questions, 
among others, might inspect the balance between an item's 
importance for equitable awards and its history of inaccuracy. 

Decisions about considering and adopting alternative larger 
or smaller sets of data for need analysis would have to be based 
on their direct and indirect effects on the program's budget, 
their redistributive effects on groups of recipients, and the 
efficiency of administrative work with specific data elements. 
The Congress in past years has rejected several Department 
proposals for simplifying the need analysis, but the continuing 
problem of inaccurate application data may call for a reconsider- 
ation of the possibility of making awards from a simpler set of 
data. 3 

IMPROVING THE FORMS 

Even if it is not feasible to prevent error by making the 
major simplifications suggested in the alternatives described 
above, it might be possible to increase data accuracy by 
improving the way applicants submit their data. Currently, all 
applicants complete and mail to a processor one of several paper 
forms, depending on the processor (see the section in chapter 1 
entitled ''how the Program Works"). In general, the forms ask 
for a variety of data in language that is idiosyncratic to the 
field of student financial-aid administration. Department and 
contractor officials told us that major changes to the federal 
application form are constrained by computer data-processing 
routines and the statutory requirement of a common application 
form for all federal student aid. 

As the statutory definition of need has evolved, and as 
various users of the data have increased their requirements over 
the years, the federal form has become more and more complex and 



^The effects of modifying the Pell grant formula could be 
calculated with data that are now available. One recent 
simulation of a much simpler formula showed that few recipients 
would lose large amounts, recipients losing eligibility under 
the alternative received low awards, and the neediest students 
would continue to receive high awards. 
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has been printed in smaller and smaller type to accommodate more 
and more questions to the applicant. (The forms supplied by the 
other processors are even more complex, since they include data 
requirements set by institutions beyond those needed for deter- 
mining eligibility for federal aid.) Dependent and independent 
students are asked to follow different color-coded paths through 
the form. Although expertise in forms design was available for 
several years as part of a subcontract with the previous proces- 
sor, the Department has used it only to a small extent. Depart- 
ment officials could recall only one small field test of alterna- 
tives to the present form. We suggest consideration of further 
alternatives in this area. 

One alternative might be to provide separate forms for 
people in different circumstances, as the Internal Revenue 
Service does with income tax returns. Radically simplified 
forms could be used by applicants who have only a few kinds of 
income to report. Some of the complexity of the present form 
results from its several questions about types of income, assets, 
and other financial conditions that may apply to relatively few 
people. A simpler form that would be useful for a large number 
of applicants might justify the cost ol including alternative 
forms in a common package. A second modification would be to 
provide one form for independent and another form for dependent 
students, to remove the confusion of the color-coded pathways. 
Some officials believe that gains in simplicity within each of 
the modified forms might be offset by the complexity of the 
instructions and questions that would be necessary to help all 
applicants deternine which form to use. This concern, and others 
such as the added costs of processing multiple forms, show that 
these simplifications and others require further development, 
testing, and analysis. Since the common application presently 
used is established in the legislation, the Congress would have 
to weigh the merits of multiple application forms at the 
appropriate time. 

The use of separate forms by different groups might make 
more space available on each type of form. Space limitations 
on the current form have prohibited several improvements to the 
application in the past. The items on untaxed income, household 
size, and number enrolled in postsecondary schools could be less 
troublesome if space permitted more detailed answers on the 
form. We noted above, for example, that untaxed income on the 
1984-85 application was to be transferred to the form from an 
aggregate of 18 items on a worksheet included in the application 
package; these as well as the household data, given now simply 
as single numbers, could all be itemized directly on the form. 
Experience in other programs suggests that accuracy improves when 
applicants not only count family members but also must list their 
names in answer to a specific question. Obviously, verification 
is easier if information is itemized directly on the form. 
Finally, making more space available by differentiating forms 
might allow more instructions to be printed directly on each 
form. Some inaccuracy in the current application may result from 



126 



ERLC 




APPENDIX VI 



APPENDIX VI 



Students' simply not reading the 11 pages of instructions that 
now accompany, but are separate from, the form. 

The Department does not know much about how applicants fill 
out the current form— when and how and with what assistance. In 

fubDect, the Department might review the experience 
Of other agencies. The housing-voucher and income-maintenance 
experiments, which tested various methods of closely monitoring 
household economics, and the Bureau of the Census might be able 
to share useful information on the conditions under which people 
KSni^., most accurate data about themselves. Knowing more 
about these conditions, the Department might then consider steps 
to create them for Pell grant applicants. st-«ps 

ri.^-. f;:^"/it^°"t major changes to the structure of the form, the 
data It elicits might be more accurate if specific data items 
were redesigned to use definitions and terms common elsewhere. 
For example, listing the number of exemptions on the federal 
income tax return might be substituted for the current Pell grant 
application item calling for household size. Many other issues 
are implicit m this proposal. For example, the language of 
^fffl! J^\^S''?f."'^^ ^ barrier; some financial aid experts 

TnlrTJ I Pell's present practice of using "adjusted gross 
income from federal income tax terminology has already had 
significantly negative side effects. However, in the search for 
S%^?n,?;^f awards more accurate, the benefit from standardizing 
the terminology might be weighed with other effects on the 
program's objectives. 

tratinn"5^^^,' technology assists more and more in the adminis- 
tration of financial aid functions, ways may be found to improve 
not only the application but the application process as well. 
For example, the direct entry of application data into an auto- 
mated information system might be possible for students already 

^ f -5°?^ and applying for a first or a subsequent 
award. The validation of data items and correction of eligibil- 
i^nv""""? ^e performed simultaneously with data entry, shortening 
many steps of the current process. If an analysis of the distri- 
bution of Pell grants were to show a concentration of awards at 
a small number of institutions, it might be possible to use 
advanced technology more intensively in administering student 
aid there, including the calculation of eligibility and handling 
of corrections after validation. The Department's current 
voluntary program of data transmission by computer tape shows 
that at least some schools take advantage of opportunities to 
benefit from advanced technology; further expansion of this 
effort could be explored. 

VERIFYING DATA 

Data are presently verified by school officials' reviewing 
documentary evidence some students provide for some application 
Items before awards are calculated. Consideration might be given 
to the selection aspects of this process, especially since the 
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congress directed 100-percent validation for all later years in a 
report accompanying the appropriation for validation in 1982-83. 
And the possibility of central rather than school-by-school 
verification might be reviewed again, even though this method has 
already been rejected several times. 

An official at one of the nonfederal processors told us 
that there is demand from the schools for centralized verifica- 
tion but, as we discussed in chapter 4, consideration of this 
option in 1981 and 1982 was brief and incomplete. Issues to 
be explored include how to protect the privacy of federal 
income tax returns that would be attached to applications, how to 
verify information from parents and students who are not required 
to submit returns to IRS, whether to require the supplementary 
schedules that some people file in addition to the basic tax 
return, and whether to continue to allow the submission of Pell 
applications before federal income tax returns have been 
completed. Central validation almost certainly would entail 
substantial costs for the processors and some burdens on the 
applicants, but these might be offset if institutions whose 
routine validation work load were reduced were able to follow up 
on exceptions and special cases more easily and to give greater 
attention to other procedures that affect the quality of awards. 

Many of the burdens to applicants, institutions, and the _ 
processor in handling paper documents could be relieved by using 
the advanced automated- information technology that permits the 
comparison of data by computer-matching. For example, it ^ might 
be possible for a centralized service to match applicants' data 
with information on income tax returns on file with IRS. 

Computer-matching raises many complex legal, ethical, 
technical, and practical issues, some of which we are reviewing 
in a separate study. Our point here is not to try to resolve 
questions about the feasibility of applying computer- matching to 
pell validation but to raise it as a possibility for considera- 
tion. Department officials generally believe computer-matching 
to be impossible for legal and technical reasons, and IRS offi- 
cials have been opposed to it. The Department has no statutory 
authority to obtain tax returns from IRS and, in any case, it is 
believed that IRS takes so long to process tax returns that effi- 
cient access could not be had to a reasonably complete central 
file until school starts late in the calendar year. This might 
be too late to affect the calculation of pell awards for many 
students, especially if the comparison of the application and the 
tax return on file revealed discrepancies requiring adjustment 
or correction. 

However, these difficulties might not be insurmountable. 
For example, it might be possible for applicants and their 
parents to grant access to their tax returns by checking an item 
on the application, or the Congress might allow access directly 
by statute, using the tax system for unrelated purposes has 
serious implications, but the Congress recently expanded the 
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permissible data-sharing in order to strengthen eligibility 
verification in several other programs. if analysis were to show 
a high probability of significant reductions in error or gains in 
efficiency, similar permission might be granted for the Pell 
program. As for the question of timeliness, irs officials told 
us that by late summer their central data files include the great 
majority of individual income tax returns for the prior year. If 
the Departm^^nt ' s data systems were improved to yield up-to-date 
records for locating at the schools applicants who have become 
recipients, and if the IRS data were made available o-. time, it 
might be possible to match data in the early fall, with the goal 
of identifying and resolving discrepancies after an award had been 
calculated and a first payment made but before the °econd 
payment. 

S UPPORTING THE INSTITUTIONS' 
ADMINISTRATIVE WORK 

Although institutions benefit from the attendance of 
students who have federal aid, they must perform many taciks in 
the application and verification processes, in the later stages 
of calculating and disbursing awards, and in monitoring recip- 
ients' academic progress, issuing refunds, and keeping financial 
records on all transactions. The Department has data on institu- 
tional errors that contribute to inaccuracy in pell awards but 
little information on the schools' practices in ar?"^.inistering 
Pell grants and other fiaancial aid programs. 

Detailed data-gathering on this *"ubject was deleted from the 
Department's 1982-83 study of Pell a./ard accuracy, even though 
little or no data had been available for gauging the added 
effort and costs of the decision in 1982 to place an expanded 
validation burden on the schools that year. By the time of the 
Department's 1983 letter to schools proposing that they validate 
applicants for other federal student aid, the Department still 
had no cost data, although the proposal required additional 
institutional tasks such as drawing statistical samples of 
student records and determining institutional error rates. 

Even with the data that are available, the Department draws 
no composite picture of its relationships with the schools, such 
as could be done by integrating data from operations, training, 
program revi' ws, and research. Because increasing the accuracy 
of Pell awards may require greater institutional effort, and 
because of the shortcomings in the analysis of the work required 
to administer federal aid, we believe special consideration 
should be given to the development of shared goals between the 
Department and schools and to alternative practices in supporting 
schools' efforts to reaqh those goals. 

The Department could consider alternatives in areas of 
cost-sharing and training. For example, other major federal 
benefit programs administered by nonfederal organizations permit 
significant cost-sharing of the expense of administration. 
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Meanwhile, officials at one federal agency told us that they 
decided not to require extensive eligibility-verification efforts 
by local agencies administering their program, explicitly because 
the agency's budget could not afford to pay them any significant 
share of the cost. The Department pays a statutory administrative 
allowance of $5 per Pell grant recipient, partly in recognition of 
t^ie work entailed in validation, but it has few data from which to 
judge the adequacy of this payment. (The Departnient has 
recormended to the Congress that the allowance be $2.") In the 
absence of Department data on the costs of administering 
financial aid, the congressionally mandat:ed National Commission 
on Student Financial Assistance studied the subject and found 
that in L981-82, before the expanded validation requirement, the 
average cost of administering a Pell grant was about $56 per 
recipient. We have estimated the average cost of each validation 
at about $14 (see chapter 2). Thus, it appears that the federal 
government shares only a small fraction of the cost of adminis- 
tering the Pell program. 

Increasing the accuracy of rc:xi awards may require a recon- 
sideration or not only financial resources but also the skills 
financial aid staff may need if new objectives and techniques 
are added to their routine. As we discussed in chapter 4, the 
Department's training for institutional officials has not focused 
to any significant extent on issues of quality control. The data 
that show inaccuracies in student applications even after valida- 
tion, as well as institutional errors of several kinds, suggest 
the value of e-^plicit training on quality-control topics for 
schools* administrative staff. Follow-up data for monitoring 
and evaluating the effects of the training should probably be 
gathered. If some institutions have developed effective methods 
of insuring accurate Pell awards, the Department might consider 
ways of supplementing training by its regional staff with train- 
ing by staff jrrom those schools. The schools that enroll the 
greatest numbers of Pell grant recipients might be allotted 
special training efforts. 

OVERSEEING QUALITY AT SCHOOLS 

Department staff visit schools participating In student 
financial aid programs to evaluate their compli=incs with policies 
and regulations (see chapter 4). Their program reviews yield 
some data on the quality of administrative processes at the 
schools, but more information and better tools for oversight are 
needed. An alternative to the limitations of program reviews 
might combine several elements: regular data from all schools, 
targets of award accuracy, and incentives for schools to attain 
the targets. Improved data on award quality, derived from 
regular operations, could be compared with the targets that had 
been set, and this in turn could be linked to incentives. Other 
federal programs such as food stamps and Aid to Families with 
Dependent Children have extensive eicperience with similar 
practices that could be adapted to the special circum'-^.ances of 
educatior.* 
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unlike Pell, some federal programs give explicit direction 
to the administering agencies about acceptable levels of award 
quality and mea^^ure them in a continuous monitoring of indicators 
of quality. Improvement plans are required when targets are not 
met. Extensive effort is made to obtain reliable data on the 
accuracy of the benefit awards made by the administering 
agencies, and regular quality-control studies of samples of 
awards at local and state levels are reviewed I j federal 
officials to verify the results. Adopting target-setting and 
monitoring practices and sharing administrative costs to a 
greater degree could put the Department in a better position to 
review with the schools the efficiency of the administrative 
processes that are carried out with the help of federal funds. 
The federal share of administrative costs could be adjusted 
upward, giving the schools an incentive to adopt practices that 
could better meet or exceed the minimum standards of accuracy 
for awards. 

The experience of other programs with comprehensive 
quality-control efforts is only suggestive, however, because the 
historical relationship between the federal government and 
institutions of higher education differs from that between the 
federal government and state and local governments administering 
other programs. Nevertheless, our outline discussion might be 
useful to the deliberations of the Congress, the Department, and 
the higher-education community if the institutions are to expand 
their effort toward making awards more accurate, if the resources 
and support are to be found for the efforts, and if methods of 
program oversight are to be improved. 



We found many ideas for improving current practices in the 
Pell grant program that deserve further study. VJays of increas- 
ing the accuracy of Pell awards include redefining the basic 
concept by which students* need is determined, refining the 
collection of the data that are used in determining need, revis- 
ing the methods by which applicants are asked to provide these 
dara^ changing the approach to the verification of data, provid- 
ing incentives to schools to improve efforts toward increasing 
the accuracy of awards, and :.inproving the resources, support, and 
oversight the Department gives to the schools that administer the 
program. Our suggestions are founded on analyses of opportuni- 
ties for improving the operations of the Pell program and on the 
experience of other federal agencies facing similar problems in 
administering programs that base individual benefits on need and 
disburse them through nonfederal, noncentralized agencies. 

The ideas we present meet our criteria of relevance and 
potential benefit, although our review did not extend to the 
gathering of conclusive evidence of the merits of the elternr. - 
tives we have discussed. Many issueo of feasibility, cost, 
technical merit, and acceptability remain to be worked out. The 
specific alternatives discussed in this appendix supplement the 
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main conclusions presented in chapter 5 on overall changa needed 
in the goals, strategies, and management of the Department's 
efforts to improve Pell award accuracy. The Congress and the 
Department could consider technical alternatives better after 
clarifying tne larger policy issue of the priority that is to 
be given to accuracy. 
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AGENCY COMMENTS AND GAO'S RESPONSE 



This appendix provides our response to major points raised 
by the Department of Education and the Office of Management and 
Budget on a draft of this report. Following our response to the 
agencies" comments, we have printed the main section of the 
Department's comments (excluding two appendixes, available on 
request) and the entire text of 0MB 's letter. 

Neither 0MB nor the Department finds factual errors in the 
draft report, but the Department interprets the e/idence differ- 
ently. The Department believes that its past performance toward 
Pell grant error has been more than adequate and that current 
and planned corrective actions constitute an overall strategy 
that will address the error that can be reduced without legisla- 
tive changes. We believe that the findings we report in chapter 
3 show that the error rate after the Department's corrective 
action in 1982-83 remains unsatisfactory. Our report notes that 
legislation simplifying need analysis could help reduce some 
kinds of error (as we discuss in appendix VI), but we also con- 
clude that the Department could do more within current law. 

The Department describes activities and plans initiated 
since we completed our review. While we agree that more is 
being done, it is more of the same kind of effort that did not 
work very well before. We believe that the continued absence of 
goals, the unclear choice of strategies, and the absence of a 
plan for improving the data still suggest the need for congress- 
ional consideration of both the general issue of giving guidance 
to the Department about correcting error and the specific issue 
of the adequacy of the Department's data on error. 

DEPARTMENT OF EDUCATION COMMENTS 

The Department organized its comments by abstracting five 
concerns from the report and responding to each one. The five 
headings below reproduce the Department's wording of each 
concern, beginning on page 144. 

"The Department lacks explicit goals 
and targets for corrective action on 
Pell Grant errors." 

The Department says that it has in operation and planned a 
broad variety of initiatives to insure that eligible students 
get financial aid when they need it. The initiatives include 
continuing validation, running a pilot test in 1985-86 of insti- 
tutionb^ expanded use of electronic data-transmission, and 
implementing other pilot projects in which institutions develop 
local quality-control techniques. In addition, the Department 
says it has begun to redesign the Pell grant application form 
and that a new form will be introduced in 1986-87. 
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The Department believes no error-reduction goals are appro- 
priate until long-term management goals have been well estab- 
lished (in part through the pilot projects of institutional 
quality control). The Department believes that it cannot make 
much progress in reducing error until the Congress gives it 
further guidance on resolving the conflict between the goal of a 
complex, need-analysis system that is sensitive to applicants' 
circumstances but prone to error and the goal of reducing 
the complexity of the system in order to reduce its errors. 
Finally, the Department recognizes that it has implemented error- 
reducing strategies without devoting adequate time and staff 
resources to analyzing and testing them in advance. However, the 
Department also says that its 1982-83 expanded validation was to 
be a quick effort, requiring institutions to use minimum re- 
sources to attain the maximum correction. The Department be- 
lieves that its continued expansion of validation shows "movement 
toward a cost/benefit perspective" (page 145) and is consistent 
with goal-oriented management. 

We agree that some of the specific initiatives the Depart- 
ment says will take effect in 1985-86 and beyond appear to be 
useful steps in improving the Pell grant program, for two rea- 
sons. First, the steps respond substantively to points we have 
identified in our analysis of potentially promising approaches, 
including the need to revise the application form to prevent 
error as much as possible and the need to address institutional 
error much more forcefully and involve the institutions in the 
process. Second, the Department appears to be committed to 
making pilot tests of some of its new ideas before implementa- 
tion, which wo have discussed as a useful approach to gathering 
information and making sound decisions. 

However, the Department's presentation of these new activi- 
ties shows a continuing absence of key elements that our analy- 
sis found were missing from initiatives taking place during the 
years covered by our data. We disagree that goals (in terms of 
the kinds of error to be reduced) and specific targets (in 
dollars and percentages) are impossible at early stages of such 
projects. These new activities and strategies are not clearly 
linked to analyses of specific kinds of error and the chances 
that each strategy can remedy those errors. it is not clear that 
the management of diverse new projects at different points in the 
Pell grant program will be more successfully coordinated among 
the divisions of the Office of Student Financial Assistance 
than projects in the past. The weak evaluation that has made 
decisions difficult (discussed in chapter 4 and appendix iv) 
seems likely to be repeated, since we do not see the Department 
discussing a plan for future data-gathering that will allow the 
critical evaluation of each new initiative. An evaluation plan 
should also be tied to a schedule of decisionmaking for long- 
term implementation to avoid the hasty decisionmaking of the 
past. 
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in suggesting in chapter 5 that the Congress consider 
offering guidance to the Department, we did not mean to imply^ 
that the absence of congressional direction is the major barrier 
to setting and achieving error-reduction goals* We agree that 
the need-analysis system presently in the law includes some kinds 
of application information that is prone to error that the 
Congress might eliminate, if reducing error is a high priori- 
ty. And we agree that the Department's dedication of major new 
resources to error reduction is likely only with the agreement 
of the Congress on the priority of the problem. But we do not 
agree that the Department can do little in the absence of con- 
gressional direction. The initiatives described in the Depart- 
ment's comments seem to argue the opposite. We continue to 
believe that the Department can set the fundamental directions 
we suggest in chapter 5 and can then design, test, analyze, and 
implement a wide range of useful ideas without further guidance 
from the Congress. We agree with 0MB that the Pell error rate 
is unacceptable, and we believe that the Department should 
seriously attempt an organized effort to reduce the rate rather 
than explain it away, await direction, or invest still more 
resources with little return. 

Finally, we are not reassured by the Department's assertion 
that its further expansion of validation is another corrective 
action. From our review of the expanded validation in 1982-83 
(reported in chapters 2-4), we cannot agree with the Department's 
characterization of that effort as a '"quick strike'" intended 
to require "minimal commitment of resources by institutions to 
produce the maximum error deterring effect" (page 145). The De- 
partment knew nothing about the likely costs of the plan, deter- 
rence is difficult to predict and measure, and even the direct 
cost-savings were significantly misestimated (as we discuss in 
chapter 4). In fact, the plan required considerable instituLion- 
al resources — 1.66 million validations at an average cost of $14 
— and resulted in a very small drop in overall error. Error was 
reduced in one of the two items that were validated (which the 
Department cites) and error was increased in the other item 
(which the Department does not mention). Expanding validation 
further without the kinds of data and analyses we suggest in 
appendix IV could result again in ineffective action, from one or 
more possible causes, including mistargeted application items, 
inadequate documentation requirements, weak selection of appli- 
cants, and uncertain evaluation of the results. 

We see no new evidence to support the Department's claim 
that further expansion of validation demonstrates "increased 
movement toward a cost/benefit perspective" (page 145). The 
Department's comment does not mention any source of new data on 
the costs of its requirements, and no such data were available 
for our review. It is difficult to see how further validation is 
"consistent with the goal-oriented type of error management advo- 
cated by GAO" (page 145), since the Department's comment does not 
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state goals for the expansion of validation and since it has 
already been shown that expanded validation reduced error very 
little* ^ 

"The Departm e nt lacks a cohesive plan 
to meet goala by developing and testing 
strategies that will coordinate and 
j.ntegrat(3! its error reduction efforts." 

The Department asks us to acknowledge that additional steps 
have been taken to reduce error since we collected our data and 
that th« Department now has an overall strategy for addressing 
the imcLngs of its quality-control studies. To support this 
statement, the Department cites several activities planned for 
the future. These include making unspecified changes in the 
processing of financial aid to integrate the management of sep- 
arate-programs, adopting a new model to guide quality-control 
activity, publishing a proposed regulation covering the verifi- 
cation of applicants' data on more title IV federal student-aid 
programs (only the Pell program now requires that applicants' 
data be verified), improving the integration of institutional 
program reviews with policy development and training design, and 
implementing the overall plan submitted to 0MB in fall 1984 for 
corrective action in future years. 

In the draft of our report, we acknowledged several of these 
steps, including the fall 1984 plan and the possible redesign of 
the application form, as steps that were taken after the close of 
our data-gathering in September 1984. However, considering these 
and the additional efforts noted in the comments, we do not agree 
that the Department has an overall strategy for responding to the 
extensive error found through its own research. We agree that 
initiatives such as those the Department lists in this comment 
could be promising, but the Department has not presented them ir. 
a way that shows how it interprets the research, chooses priori- 
ties among types of error for corrective action, and matches 
strategies with the problems chosen for attack. Thus, the 
Department gives us no basis for concluding that it has remedied 
the problems we discussed in chapters 4 and 5. 

For example, during our review, we interviewed Department 
officials about the project to develop an "Integrated Student Aid 
Delivery System" (page 146). The work has been under way for 
some years, but it is directed by Department staff who are 
separate from the Office of Student Financial Assistance, and 
o^ricials m the latter office knew little about it and could not 
tell us how it might address Pell error. The quality-control 
methodology that the Department describes in its conunents seems to 
oe a useful model for making decisions and taking action (involv- 
ing goals, trials of solutions to problems, implementation, and 
evaluation), much as we have encouraged the Department to use. 
However, in the Department's general discussion of quality con- 
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trol, we could not discern concrete details of what adopting the 
ideas in the model will mean in practice or how error will be 
any more correctable • 

A second example is the proposed regulation that would 
expand validation requirements to include applicants to other 
campus-based federal-aid programs. The Department has only pre- 
liminary data on errors in the campus-based programs; the first 
large-scale research on the subject is still going on and was 
recently delayed a year. Data from applicants to other programs 
probably have the same sorts of error problems we found in the 
Pell program, because the other campus-based programs share the 
same application forms, but payment consequences can differ if 
the need-analysis formulas for the other programs weigh the 
application items in different ways. Institutional error may 
also differ, because institutions have a greater role in the 
other prograiPvS than in Pell. Therefore, applicant validation is 
likely to be only a partial strategy against overall error in 
campus-based programs, and it is being implemented before the 
Department knows the full scope of the problem. 

Finally, we agree that training and policy development could 
be more relevant tools for correcting error if they reflected 
better the results of program review. Much more important, how- 
ever, would be linking training, policy development, and program 
review with what is known about error from the national research 
studies. In chapter 4, we note a series of weaKnesses in the 
Department's use of training and program review as strategies 
against the error described in the research. These weaknesses 
should be strengthened rather than making wider use of data from 
unsystematic program reviews. 

We have acknowledged that late in 1984 the Department sub- 
mitted to 0MB a new plan for dealing with error and that early 
in 1985 0MB told the Department to carry it out. It remains to 
be seen whether the Department will create the conditions for its 
successful implementation, including publicly stated goals and 
targets, strategies clearly linked to the chosen goals, strength- 
ened internal management arrangements that permit clear lines of 
authority for error reduction, and plans for improving the data 
that will be used to assess the next round of corrective action. 

"Problems of strategy include major 
information gaps which hamper planning 
by leaving sources of problems and 
reasons for error unidentified 
or unexplained." 

The Department first agrees that there is room for improve- 
ment in gathering and evaluating data and then describes the 
Department's efforts to use data from the application processor 
to find applications that seem erroneous and to track eligibil- 
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ity changes after suspicious patterns have triggered selection for 
validation and subsequent correction. The Department also 
describes the use of error-prone modeling in selecting applica- 
tions for validation. Finally, the Department says that an 
automated system will be used to keep track of data on institu- 
tions "infractions and management weaknesses" (page 151) found 
in program reviews and that this will help in setting priorities 
for assigning staff and in understanding the relationship between 
the causes and symptoms of institutional error. The Department 
says that the problem is "a surfeit of data" (page 152), rather 
than data gaps, caused by the large amount of information the 
Congress requires for determining students' financial need. 

We agree that the Department has a great deal of data, but 
we disagree that they result particularly from the complex need- 
analysis system. The difficulty we find is that Department 
managers do not have the data they need and must therefore make 
do with weak substitutes, including proxy measures and personal 
impressions from chance encounters. For example, the plan to 
enter data from program reviews into a computerized data base has 
limited utility when the data come from an unrepresentative set 
of schools using an unsystematic sample of records reviewed 
according to an unstandardized protocol. Such data will not show 
the symptoms of institutional error and their causes, because the 
review is not designed to produce a reliable measurement of error 
or data on causes. 

The Department does not address our major points on its 
data gaps and limited analysis. These points include (1) the 
episodic and incomplete error data (chapter 3), (2) the weak 
data used at key points in setting the parameters of validation 
and in evaluating it (appendix IV), (3) the limited knowledge of 
institutions' activity in administering federal student-aid pro- 
grams (chapter 4), and (4) the lack of information about practi- 
ces in other agencies facing similar problems of quality control 
m administering programs that base aid on need (chapter 4). 
The Department's comments do not describe a significant program 
of data improvement to resolve any of these issues, although the 
Department notes (in an appendix) its general intention to study 
other agencies' practices. 

"The lack of a central direction of policy on 
error and fragmented management responsibilities 
within the Office of Student Financial Assistance 
have led to confusion over responsibilities 
for acting upon error problems . " 

The Department believes that it has made considerable prog- 
ress in addressing this concern, even though it also says that 
we did not adequately study the Department's methods of deter- 
mining and coordinating policy. The initiative that the Depart- 
ment believes will best resolve the matter is the establishment, 
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since our review, of a new organizational unit responsible for 
quality control, the Debt Collection and Management Assistance 
service (page 152), which reports directly to the assistant sec- 
retary for post8econdary education. The Department says that the 
new unit will be in a good position to develop policy on such 
topics as pell grant error and will also be accountable for the 
direction, clarity, effectiveness, and measurability of error 
reduction. The Department also cites the Secretary's Student Aid 
Task Force (page 152), which it says will be developing a compre- 
hensive strategy for improving financial assistance. 

We disagree that we failed to consider fully the Depart- 
ment's methods of setting policy on Pell error. In our review. 
Department officials and outside observers with firsthand exper- 
ience of the Department's decisionmaking agreed that all the 
existing processes had not consistently yielded timely and well- 
analyzed decisions on error and its correction. 

We agree that structural change to clarify responsibilities 
and to place quality control at a high organizational level is 
potentially useful, especially since we have concluded in chapter 
4 that unclear responsibility has led to inaction and flawed 
decisions. However, establishing an office is not by itself 
sufficient. Quality-control experts told us that effective work 
requires clear commitment at all levels of an organization. In 
addition, a new unit needs resources to carry out its assigned 
tasks. Throughout the period of our review, we saw only modest 
evidence of top-level commitment to the problem of Pell grant 
error, and public discussion by Department officials of the re- 
search data and corrective actions has been extremely limited. 
It is not clear that the new unit will be perceived as having a 
mandate to direct the work on the problem that other established 
units perforru. And, in light of the assignment of responsibility 
to the same unit for another longstanding problem, that of col- 
lecting defaulted student loans, we question whether the re- 
sources will be available for the work that will be required to 
develop, coordinate, and oversee new ideas for the correction of 
pell errors. Staff who were responsible for quality control 
before the establishment of the new unit and who will form a key 
part of it told us that they were frequently called to work on 
difficult problems throughout the Office of Student Financial 
Assistance, making it hard to sustain work on Pell error alone. 
This difficulty could continue, or intensify, if the unit has 
major responsibility for collecting defaulted loans. 

Nor is it clear to us that the task force will be able to 
supply the overall direction that has been missing. This group 
has a wide-ranging assignment to consider a variety of issues on 
the management of student grants and loans, raising the question 
of when specific guidance for the Pell program can be expected. 
If the Office of Student Financial Assistance does succeed in 
strengthening its methods for making and carrying out decisions 
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on Pell error, there is a potential for overlap and lack of 
coordination with the task force. 

Therefore, we agree that the new unit and the task force 
may be useful, but we believe that the conditions for their 

^^^^ ^"^ resources— may not have been 

avoided. We see no reason to modify our suggestion that the 
Congress may wish to give the Department further direction on 
the subject of Pell grant error. 

"The Department lacks reliable forecasts 
of the remedial effects of its validation 
strategy and n eeds to move away from making 
decisions under tight time constraints 
towards a more analytic approachT^ 

The Department believes it is appropriate to devote greater 
resources to developing new criteria for selecting student ap- 
plications for validation. Part of the research needed to do 
this IS included m a task that is expected of the current ap- 
plication-processing contractor: to deliver an annual "require- 
ments analysis," or review, of the past year's validation effort 
and a prediction of the effects of possible alternatives for the 
coming year. In addition, the Department says it will ask bid- 
ders on the next contract to integrate the annual file on appli- 
cants with a separate file on payments to recipients that is 
maintained by another contractor. 

Concerning the more general problem of timely decision- 
making, the Department refers to the set of current initiatives 
coii-ctively and suggests that through them the "evolutionary 
rath.-r than sporadic nature of the Department's decision-making 
will become increasingly apparent ..." (page 154). The Depart- 
ment says also that analysis for decisionmaking will shift from 
the measurement of error to the evaluation of its correction. 

Regarding improvements in validation, the Department offers 
two additional points beyond those outlined earlier in its com- 
ments. The first concerns the "yearly requirements analysis" 
(page 153). It is not clear precisely what this analysis will 
consist of, but its ambitious objectives include (1) an examina- 
tion of the current performance of validation selection methods, 
(2) an examination of alternatives for the future, and (3) the 
development of a specific "selection strategy for the subsequent 
award period" (page 153). In the absence of any detail on how 
these will be accomplished, we remain skeptical, given especially 
that all the necessary data to examine performance and to develop 
a full range of selection criteria (such as criteria that are 
based on error) are not available to the application-processing 
contractor. Improving selection methods and other aspects of 
validation will require an integration of the various kinds of 
data that we discuss in appendix IV. 
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The second point on validation is about a plan to merge the 
data on applicants and recipients. Although this could help 
strengthen the Department's analyses of the effects of valida- 
tion and would be essential if computer-matching were to be used 
to combat error, a combined data system is apparently only a 
long-term possibility. The Department states only that it 
intends to "request offerors to devise a system" (page 154) for 
merging data when the next processing contract begins in 1987-88. 

Neither the Department's point on the improvement of 
decisionmaking generally nor its other comments provide us with 
enough concrete discussion of improvements in such things as 
goal clarity, management structure, and data availability to 
change our conclusion that the Department needs a fundamental 
review of its approach to Pell grant error, aided by congress- 
ional consideration of the issue. The Department's future eval- 
uation of corrective actions will be useful but no substitute 
for measuring and monitoring error. Nothing in the Department's 
comments suggests a clear commitment or a plan to continue and 
improve data on a basic element of policy — student and school 
error rates. These data could be gathered by institutions' 
studies of their own processes, by studies of national samples, 
or by other means, but no method is specified in the Department's 
comments. No improvement in decisionmaking is likely without 
timely, credible, and detailed data on error and its causes. 

QMB COMMENTS 

(MB concurs with our findings and believes the report will 
be useful. 0MB notes particularly that it has shared responsi- 
bility for the Department's quick response to the evidence of 
1981 Pell error rates and that there was and still is much to 
learn about how and why errors occur. The error rate in the Pell 
program is unacceptable, according to 0MB. 0MB reports that it 
will continue to take an active role in working with the Depart- 
ment on Pell grant errors, including a sharing of information on 
the promising practices of other agencies. 
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UNITED STATES DEPARTMENT OF EDUCATION 
OFFICE OF THE ASSISTANT SECRETARY FOR POSTSECONDARY EDUCATION 



Mr. Richard Fogel, Director 
Human Resources Division 
U.S. General Accounting Office 
Washington, d.C. 20548 

Dear Mr. Fogel: 

Tne Secretary asked tnat I respond to your request for coirments on your draft 
report entitled, »'Pell Grant Validation Inposes S(xne Costs and Does Not 
Greatly Reduce Award Errors: New Strategies Are Needed." 

The enclosed coninents represent the tentative position of the Department and 
are subject to reevaluation when we receive the final version of tnis report. 

Ife appreciate tl-:e opportunity to meet with your staff to clarify the draft 
report before submitting our response and the opportunity to conment on the 
draft report before its publication. 

Sincerely yours. 



Edward M. Elinendorf 
Assistant Secretary 



Enclosure 
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^fe nave reviewed the General Accounting Office report on Pell Grant 
Validation, and we generally agree with the findings with only one. 
qualification: they are a snapshot of efforts that existed for the 19B2-bi 
Pell award year and, for obvious rensonsi do not reflect the nurnerous 
DepartiTvental initiatives either already underway or in advanced planning 
stages, wnich would fully address the issues and concerns raised in the 
report* Wte would also further point out that ED and GAO seem to agree in 
principle tnat some of tne concerns and issues raised in the report result 
from systemic proDlems in a program that has undergone enormous and rapid 
growtn since its first award year, 1973-74; these systemic problems may be 
corrected only by correcting the system that created them and will never be 
corrected by verification alone. As we discussed in some detail with you in 
our meeting June 5, many of the concerns you raise are further complicated by 
two additional factors: 

• tne Department has tne difficult task of balancing sensitivity to 
differences in applicant circumstances against the resulting coipplexity in 
the applicrtion process that introduces a greater potential for student 

error (See Vl-2 through VI-5); and PP* 123-25 

• the Department is faced with the task of addressing the systemic problems 
while continuing to efficiently operate the programs. 

Given the system in which we have to operate, we believe that our performance, 
was more than adequate. Since the time of your report, we believe even more 
strongly that ED nas initiated several major projects that will satisfactorily 
address the concerns and issues GAO raises. The Department is designing a 
major initiative, known as "ISADS," aimed at providing coordinated managems-.c 
in all Title IV programs. The Department has also created an additional 
administrative office, the D^t Collection and Management Assistance Service 
(DCMAS), with oversight a*, coordination responsibilities, that should prevent 
any recqrrence of the issues you have identified. DCMAS includes a Division 
of Quality Assurance and a Division of Systems Design and Development to 
provide tne adninistrative structure for bringing to bear the latest 
technological advances in ADP and ongoing quality measurement efforts to 
simultaneously examine Pell problems globally and operationally. Furthermore, 
this unit will examine student aid programs generally, as they are currently 
structured and as they might be restructured, to eliminate the problems that 
have developed, probably inevitably, from the historical factors that have 
similarly figured in the expansion of the Pell Grant program. Additionally, 
OPE and OSFA are adopting, as part of the management process, a "Quality 
Control Cycle" for establishing, monitoring, testing, and reaching program 
goals. These initiatives are discussed in detail in the body of the response 
and listed in an appendix to the response. 
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In developing our response to your draft report, we ftave identified five major 
concerns and responded to those in detail. In Appendix A, we also respond to 
all otHer concerns that were menticiied in the report but were not its focus. 
And finally, in Appendix Q, we have made a list of the ED initiatives that we 
discuss m our response and provide you with the names and telephone nuiubers 
of contact persons should you wish additional information about any of these 
initiatives. 

The Department appreciates GAO's recognition of the major task we face. We 
invite you to share with all other interested parties the systemic nature of 
the problem and the limited range of solutions for correcting it. As long as 
legislation leading to program simplification is lacking, the Departront must 
devote significant energies sinply to progra^n administration that it otherwise 
could direct towards the issues brought up in this report. 



MOST SIGMIFICAOT OOMCSRMS 



1) GAO Concern: The Department lacks explicit goals ana targccs for 
corrective action on Pell Grant errors. 

ED Response: v-ftiile the Department has not laid out explicit goals for 
step-by-step error reduction in Pell Grant awards, as the 
GAO report acknowledges (p. 4-3) the Departirent has ?• 67 
undertaken a broad variety of initiatives with the 
objective of assuring that needy students receive financial 
assistance when they need it, as provided by the 
appropriate acts of Congres.s. Many of the problems pointed 
out by GAO as stemming front the ab^^nce of explicit error 
reduction goals result from systemic factors affecting 
eligibility and awaru u*. >.ribution, which render such goals 
meaningless without an accortpanying reform of program 
re-structure. Even the GAO concedes that certain 
application items for whi.^h current governiuj legislation ^ cq 
requires answers are inherently prone to error (p. 3-10, ' 
i:;, or difficult to corroborate (p, IV-5), and that need P» 
analysis could be simpler (p. vi-2), with less data p. 123 
required from applicants (p. Vl-t) . liowever, without p 125 
further Congressional guidance, the Department alone cannot 
take long or quick strides to resolve the tradeoff between 
sensitivity to an applicant's circumstances and reduced 
program complexity, that achievement of explicit error 
reduction goals would require. In particular, efforts 
towards redesigning student aid programs have suffered 
defeat in recent years, as Congress has mandated management 
to maintain the status quo *a deteiMining eligibility and 
making awards. 

Beyond the limitations on management inihiAtive posed by 
restrictions inherent in the current program structure, the 
imn:iediate need to reduce error has conpelled tlie Department 
to implement- remedial strategies without devoting adequate 
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tune aiid staff resources to analyzing or testing tt:em in 

advance. For 1982-83, the strategy undertaken by the 

Department was in direct response to pressures resulting 

from 1980-81 error studies, and was aimed as a "quick 

strike" needing minimal coitmitment of resources by 

institutions to produce the maximum error deterring effect. 

Although this effort was successful to sane extent, the 

Department did not plan the method of selection or the 

limited nature of the items checked as a continued, 

long-term strategy. In the three award periods since then 

for which we have processed applications, validation 

selections have continued at similar levels, and the 

application items subject to proof have expanded to include 

the top four sources of student error identified by GAO 

(Table 3-3). This demonstrates the Department's increased table 20 

movement toward a cost/benefit perspective on strategy 

development, that is more consistent witn tlie goal-oriented 

type of error management advocated by GAO. 

Moreover, the absence of tne incremental quantitative sort 
of goals, which the GAO seemed to find wozt appealing, does 
not justify the conclusion that the Department lacks 
meaningful overall goals for improved program management, 
or that management has failed to take more imrtvediate 
initiatives to achieve them. For 1985-8b, the Department 
has already launched an electronic data submisbion pilot 
project tnat provides for increased interaction between 
students and their financial aid administrators, a 
reduction in unsolicited and likely inaccurate application 
corrections, the capability for automated reference to 
proper cost and enrollment data to reduce the likelihood of 
institutional error in award determination, and an eventual 
potential for aliasing an electronic mode of application 
submission, as contenplated by GAO (p. VI-10) . Furthermore, 
for 1986-87 • the Department wi 11 i ntr oduce a redesigned 
application form that separately addresses independent and 
dependent stuaents, so that applying will be a less ccnnplex 
task and unintentional error arising from applicant 
confusion about dependency status will diminish. 

As another major step forward, the Department is now 
developing a pilot project to make the institution, as the 
critical delivery point in student aid programs, the focus 
of quality control. This pilot project lays the groundwork 
for long-term management goals that include: 

• enacting a comprehensive scheme for controlling error in 
an institution based on its error rates; 

• developing error measurements in applicant reporting and 
award computation that lend themselves to 
institutionally specific and readily tracked corrective 
action plans; and 
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« establishing a national baseline for performance. 

As these goals translate into functioning activities, 
establishing quantitative cortparisons for rating their 
success becomes feasible and useful. However, until 
current initiatives become CfStablished procedures, 
process-oriented statements of goals are more appropriate 
as milestones than bottom line measurements of overall 
error reduction. 

Thus, wfe believe we have in fact developed explicit goals 
for corrective action on Pell Grants that continue and 
expand the validation effort, while also supplem^ting it 
with tne imaginative and coordinated efforts that 
constitute a valid "global" policy. The Pell electronic 
pilot project, the institutional quality control pilot 
project, the establishment of the Office of Debt Collection 
and Management Assistance Service (DCMAS) in the Office of 
Postsecondary Education (OPE) , and the intensive efforts to 
integrate and systematize all these efforts, represent a 
staff effort both to correct systeTiic problems and to 
continue to efficiently administer this program, using the 
matrix of legislation and regulations that have developed 
throughout its history. 



2) GAO Concern: The Department lacks a cohesive plan to meet goals by 
developing and testing strategies that will coordinate and 
integrate its error reduction efforts. 



Response: The Departront is in the design phase of a major initiative 
aimed at providing coordinated management in all the Title 



ly programs. This project, known as the Integrated Student 
Aid Delivery system (ISADS), is a ^oint effort of the 
Department's Office of Management and the Office of 
Postsecondary Education. The goals of the project are to 
iniorove delivery, coordination, efficiency and information 
systems. iSADS provides the opportunity to identify and 
correct deficiencies in the programs in a systematic way 
and has received widespread Departmental su£^rt. 

Additionally, OPE and OSFA are approaching the irtpleraenta- 
tion of these and other error reduction initiatives 




implementing solutions. The management of the applicant 
error rate will consequently inprove, we believe, in the 




ways: 
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• monitor the overall performance of the system for 
collecting, processing, editing, selecting and verifying 
applicant information; 

• ensure the accuracy of data used for distributing aid; 

• ensure the proper execution of formal procedures; 

• identify points in the chain of student aid delivery 
where error is liKely; 

o provide managanent with regular feedback on the quality 
and efficiency of the various subsystems; and 

• provide a means for developing, testing, and analyzing 
proposed corrective actions. 

By iirplementing the quality control features of the "QC 
Cycle" as an ongoing activity within OSFA, we envision that 
problem areas will become readily identifiable and 
correctable there. This approach will also enable the 
Division of Quality Assurance to independently monitor 
OSFA ' s qual i ty control processes and to check for 
aberrations and failures of that system, as part of its 
responsibility for the quality assurance feature of the "QC 
Cycle.-' 

In order to furtner coordinate error reduction efforts as a 
unified approach, the Department is planning to publish a 
proposed regulation that would expand validation to more 
Title IV programs in an overall effort to reduce error. 
This proposal will help assure that aid under the 
Campus-Based programs is properly distributed and that 
Guaranteed Student Loans (GSLs) are made only to eligible 
students. By pro/iding a single, integrated verification 
system for the Pell Grant, Campus-Based, and GSL programs, 
the Department hopes to bring all potential recipients of 
Federal student aid under a process that provides 
reasonable assurance of accuracy in data used to determine 
eligibility, without unduly burdening institutions with 
compliance checking procedures specific to each Title IV 
program. As a further improvement, this proposal addresses 
the GAO concern that the margin of allowable error in 
applicant reporting may be excessively loose (p. IV-6, 7), f 
by allowing no more than a ^100 net difference between 
documented and reported values for all application items 
before an eligibility redetermination is necessary. 

There are other examples of coordinated efforts that 
demonstrate tne degree of interaction between various 
program related activities and challenge the GAO assertion 
about the absence of a cohesive plan. First, differences 
in the aggregate amounts of institutional error found in 
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the studies done for 1980-81 and 1982-83 are largely 
attributable to the regulatory changes and refinen-^nts that 
nave arisen from a longstanding effort to cx>ordinate the 
results of program reviews with policy developnnent. 
Moreover, with the introduction of the Pell Grant quality 
control pilot project, the involvement of staff doing 
program review will cover both developing regulations 
governing new institutional procedures ani devising methods 
to integrate successful institutional efforts to reduce 
error, with institutionally perceived program review 
benefits. Other recent regulatory changes, resulting from 
the interaction of different bu'v related Department 
functions, that demonstrate an integrated approach to 
properly channeling student aid funds under institutional 
stewardship include: 

• limiting the amount of remedial course work for which a 
student may receive Title IV funds; 

# requiring the return of payments made to students who 
never attended any classes; and 

o making class attendance a part of the definition of 
"enrollment." 

Secondly, while the Department's institutional training 
will continue to offer programs for disseminating 
regulatory and procedural changes, it will broaden its 
responsive emphasis to provide targeted training efforts 
that address specific institutional deficiencies, wnen 
program reviews find that areas of institutional 
misunderstanding or lack of information are the source of 
the problems detected. However, the benefits of this 
approach may be compromised, since in the absence of new 
legislation, the Department lacks the authority to require 
tne participation of institutional personnel in training 
activities, regardless of how great the Department's 
appraisal of their need for it may be. 

In conclusion, contrary to GAO's assertion, the Department 

has developed a cohesive strategy for reducing the error 

rate in tne Pell Grant Program, as evinced by the short and 

long-terra corrective action plan suhtiitted to the Office of 

Management and Budget and recognized by GAD (p. 5-11). Some p. 90 

of the proposals for corrective action will commence with 

the 1985-86 award year. Other proposals will require more 

time to implement because of necessary regulatory changes, 

or because of changes to the authorizing legislation which 

need Congressional consent. 
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Overall, while we believe there is always room for 
inv)rovQT^nt, we request GAO to acknowledge in their final 
report tnat the Department has taken additional steps to 
reduce error, and does in fact have an overall strategy for 
addressing the findings of its quality control studies. 



3) GAO Concern: Problems of strategy include iTiajor information gaps which 
hairper planning by leaving sources of problems and reasons 
for error unidentified or unexplained. 

ED Response: While we agree that there is room for utprovement in 



gathering and evaluating data, the Department has 
consistently reviewed a substantial body of information 
since the incepticw of validation in 1978-79. Although tne 
Department lacks many kinds of data that would be most 
appropriate for attributing error to specific causes, it 
has attenpted to corrpile information from the application 
processing system that can allow identification of 
suspicious daua and measurement of the effects on 
eligibility arising from correcting it. The system of 
edits used to check information supplied by applicants for 
consistency and reasonableness has undergone evaluation 
several times by independent contractors and the 
applicaticxi processing contractor, and has demonstrated its 
effectivQiess as a mechanism for eliciting corrections to 
r^)orted information. In addition, a recently developed 
system for management information reporting has provided 
regularly produced tables and graphs that further extend 
the Department's knowledge about the correction benavior of 
students having suspect data, but who are not selected for 
validation. This system allows for data conoparisons within 
and across award years and presents projections derived by 
the Department, so that current data appears framed 
retrospectively and prospectively. 

For each year since the Department began selecting 
applications for validation, it has gathered information 
during the processing year, as well as in special studies 
prior to each year's development of criteria, to determine 
the affect on eligibility resulting from changes made by 
the applicant subsequent to selection for validation* The 
selection criteria are then evaluated to determine and rank 
their effectiveness in selecting those applicants who are 
likely to nave misteported. Since 1983-84, the adoption of 
validation status codes, on the payment documents submitted 
to the Department by institutions, has permitted the 
analytical capacity for determining the degree to which 
selection by each criterion has resulted in detecting data 
requiring correction or within allowable accuracy 
tolerances. As the development of the new Pell Grant 
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payment processing system progresses/ the Department will 
be increasingly able to use this information which so far 
it has been unable to organize, because of problems in 
system programming arising from the novelty of the process. 

With the exception of 1982-83, the Departm^t has developed 
new criteria for selection and revised existing criteria to 
in^rove targeting. In the past two years, we have 
implemented error prone modeling techniques which use 
applicant characteristics to predict which groups are 
likely to have made errors. Although the Department had 
planned to adopt at least partial results of this analysis 
technique for use in 1982-83, it did not follow through on 
iitplementation that year for two primary reasons: 

1) at that time error in the program, based on the Quality 
Control Study of the 1980-81 award year, seemed so 
widespread that the only adequate response was to 
select all applications, at least for a portion of the 
processing period; 

2) the error prone model developed was still considered an 
experimental approach, which needed an additional year 
for further refinement and evaluation. 

By the time 1983-84 processing began, development and 
evaluation were completed, and this selection technique was 
used to supplement existing criteria based on the 
Department's experience. 

Error prone models significantly oihance selection 
par ticularly in conpar ison to random sampl ing , and 
represent a step forward in targeting applications that 
need further review. The Department has used error prone 
models as a tool to expand validation selection on a 
targeted basis for three years, and each year has refined 
and improved them. 

From the standpoint of detecting institutions whose 
practices necessitate greater oversight, program review 
staff are developing an autonated system which will provide 
summary data on a number of aspects of institutional 
stability, including its history of infractions and 
management weaknesses. Tnis will allow improved monitoring 
of institutions with a high likelinood of administrative 
deficiencies, and provide a mechanism for setting 
priorities in assigning staff. By this means, the 
Department should attain a better understanding of the 
relationsnip between the causes and syrpptoms of 
institutional error. 
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Again/ ED attributes the major cause for this concern to 
systemic problems in the Pell Grant Program. The mandate 
to ensure equity in this need-based program has in past 
years resulted in a longer Federal application, so that on 
occasion the problem has not been the information gap you 
cite, but a surfeit of data that makes analysis and 
interpretation difficult. We believe that the initiatives 
we discuss here and elsewhere in the report resolve GAO's 
concerns and represent a major step in the systematic and 
thoughtful gathering, analysis, and reporting of Pell Grant 
Program data, including data about validation. 



4) GAO Concern: The lack of a central direction of policy on error and 



fragmented management responsibilities within the Office of 
Student Financial Assistance have led to confusion over 
responsibilities for acting upon error problems. 



ED Response: Of all the concerns raised by GAO in its report on Pell 



Grant validation, we believe we have made the most progress 
in redressing this concern. Before listing the initiatives 
that justify this statement, however, we would like to 
point out that there have always been both routine and 
issue-oriented mechanisms for determining policy and 
coordinating management. Within the Office of 

Postsecondary Education, these include weekly '^^tings of 
all division directors, inter-divisional task forces to 
address particularly formidable tasks, and the Office of 
Policy Development. We believe, again, that GAO has reached 
Its conclusion using facts that it has derived from the 
nature of systemic problems witl^ the Pell Grant Program, 
rather tnan from a conprehensive review of the mechanisms 
for policy making and coordination. 

ED continuously examines its policy making and management 
accountability processes. The establishment of the Debt 
Collection and Managanent Assistance Service in the Office 
of Postsecondary Education, which is one result of this 
constant self-evaluation, now provides the framework for 
even more carefully coordinated development of policy 
related to error reduction in the Pell Grant Prograia and 
for a clearly identifiable management structure accountable 
not only for central direction of efforts, but also for the 
clarity, effectiveness, and measurability of the error 
reduction process, GAO is also aware of the Secretary's 
Student Aid Task Force, on which the Assistant Secretary 
for OPE serves. The work of this Task Force will further 
intensify our efforts to manage the programs efficiently 
and effectively by reviewing past efforts to achieve this 
goal and developing frcm this review a comprehensive 
strategy for the delivery of Federal student financial 
assistance, including Pell Grants, to eligible students. 
The Task Force and DCMAS alone constitute significant 
Departmental efforts to reduce error. 
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However f in our three responses to your major concerns that 
precede this discussion, ED has also discussed other major 
initiatives eitner underway or in the advanced planning 
stages that reflect careful direction of policy, 
coordinated cnanagement of error reduction activity, and 
identifiable and accountable management authority. These 
include tne Secretary's student Aid Task Force, the 
Integrated student Aid Delivery System Task Force, the Pell 
Grant Electronic Pilot Project, the Institutional Quality 
Control Pilot Project, the "UC Cycle" shown in Figure 1, 
the proposed Integrated Verification Regulations, the 
increase in the number of items the Department requires 
each institution to validate, the redesigned Application 
for Federal Student Aid (AFSA) used by Pell Grant 
applicants, and the continuing review of the clarity and 
coiT¥)leteness of botli the Federal form and accon¥>anying 
instructions. 

All of tnese initiatives directly or indirectly affect the 
Pell Grant Program and increase the Department's capability 
initially to detect and correct error, and ultimately to 
reduce error to a level that will ensure that the goals of 
tne Program are being met and that will reinforce the 
public's confidence in the manageinent and integrity of tne 
Pell Grant Program. 

5) GAO Concern: The Department lacks reliable forecasts of ttie ratkxlial 
effects of its validation strategy and needs zo move away 
from making decisions under tight time constraints towards 
a more analytic approach. 

ED Response: As stated in preceding responses, the Department has 
maintained an ongoing analysis process tnat has laid a 
statistical foundation for justifying and evaluating the 
validation strategies that it has already adopted. Beyond 
tnis effort, the Department believes devotion of greater 
resources to developing new validation selection criteria 
IS appropriate. As an integral aspect of the "QC Cycle" 
described previously, this data collection and organization 
will increasingly address long range goals and concentrate 
on redirected analytical functions. 

An iitportant part of this ongoing research is now included 
as an annual deliverable under the current application 
processing contract. This yearly requirements analysis 
serves as the basis for developing a selection strategy for 
tne subsequent award period, and features an examination of 
system performance for the current year using a sairple data 
base, as well as a predictive capacity for evaluating the 
performance of proposed selection criteria. Its results 
will be sufficient in breadtn and scope to allow effective 
error rate management and eventual reduction, because it 
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can cover examination of as yet uniirplemented approaches, 
including the selection and verification of applicant data 
used to establisn eligibility for other Title IV programs 
besides Fell Grants. Aside from forward looking analysis, 
our contract for processing applications also provides an 
inproved method for accessing and analyzing current data 
througn botn an on-line information center and the 
availability of "ad hoc" reports, which can be produced 
irregularly and designed to suit specific inmediate 
information needs* 

In order to remedy the analysis problems caused by the 
difference in timing between receipt of application data 
and receipt of corresponding payment data, the Department 
intends in its next RFP for Pell Grant processing to 
request offerors to devise a system that will integrate 
both functions and provide a cortplete record from 
application to payment. Such an integrated system is made 
possible by a revised design of the Student Aid Report that 
allows for automated payment processing, faster 
institutional account reconciliation, and incrementxil 
adjustment of institutional allotments during the current 
award period. When combined with other initiatives that 
will enhance development of validation selection criteria, 
the Department will achieve a greatly iirproved capability 
for accurately predicting program savings due to its error 
reduction efforts and for adjusting selection paran«ters to 
meet established goals. Consequently, the Department could 
then balance selection volume with its concomitant burdens 
on students and institutions in an objective manner along a 
formally established procedure. 

As current initiatives become established procedures, the 
evolutionary rather than sporadic nature of the 
Department's decision-making will become increasingly 
apparent, with the gradual shift in its analysis focus from 
error measurement in existing functions to corrective 
action evaluations through pilot projects and field tests, 
that will allow experimentation without disrupting service. 
The newly created organizational structure within OPE but 
distinct from the program administration structures that 
was described earlier, demonstrates the Department's 
commitment to sound and continuous oversight practices, and 
serves as a mechanism for assuring adherence to a rational 
aijd systematic approach for achieving error reduction. 
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EXECUTIVE OFFICE OF THE PRESIDENT 
OFFICE OF MANAGEMENT AND BUDGET 



WASHINGTON. O.C. 20S03 



Jim 0 5 1985 



Mr. William J. Anderson 
Director 

General Government Division 
United States General 

Accounting Office 
Washington, D.C. 20548 

Dear Mr. Anderson: 

Thank you for the opportunity to comment on GAO's draft report 
on verification of applicant data in the Pell grant program 
(PEMD-85-lOf dated 4/29/85). We appreciate GAO's extension of 
the time available to us for review. 

We generally concur in the report's findings and believe its 
lists of "matters for consideration" to be thoughtful and 
appropriate ones. The detailed information in the report is 
extremely interesting and will be of considerable use both to 
0MB and the Department in making further progress on this 



Let me say clearly that 0MB shares responsibility for the 
Department's recent substantial increase in verification 
requirements. Both we and the Department realized when error 
rate data became available that there was (and still is now) 
much to learn about how and why errors occur in these programs, 
but error rates were so high that some immediate and decisive 
management response was clearly required. The Department, with 
our approval, took what seemed to be the most sensible 
short-run steps at the time. Since then, the Department has 
taken a number of additional steps to learn more about the 
problem and to improve our ability to address it more 
effectively, while continuing stringent validation 
requirements. We believe that this is an appropriate approach 
with programs afflicted with such unacceptably high error 
rates. 

With regard to specific report comments directed to 0MB, we 
appreciate GAO's support of our efforts to assist the 
Department in reducing Pell errors. We fully intend to 
continue an active role, including supporting expansion of 
validation to other student aid programs. In this regard, we 
are particularly concerned by GAO's finding that there is 
limited feedback from ongoing program review activities to 
either improve quality control actions (p. 4-10) or revise 
training curricula (p. 4-12) . We intend to review this matter 
further with the Department. 
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We appreciate and share GAO's concern about the need for 
further sharing of information across agencies on quality 
control* In this specific instance, we have put ED staff in 
touch with Labor Department staff who have had recent 
experience in mounting a comprehensive quality control program. 
In addition, examiners with responsibility for programs with 
established quality control systems, like APDC, are assisting 
our Education Branch in developing quality target approaches of 
the sort GAO suggests in pp. 16-18 of Appendix VI; the 
Department is also looking into such approaches. More 
generally, OMB is actively promoting the exchange of promising 
practices among Federal agencies to reduce payment errors 
through the Payment Integrity project. Under this Reform 88 
initiative. Federal agencies are working together to develop 
and implement better verification methods, computer matching 
techniques, and quality control systems. OMB will continue to 
facilitate and improve the development of successful error 
reduction strategies for the Pell Grant program through this 
initiative. 

Thank you again for the opportunity tc comment. We regard this 
report as a useful contribution to addressing a major 
management problem and look forward to its publication in final 
form* 



2 
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Sincerely, 




(973189) 



156 



ERIC 



177 



